Generating augmented reality images using sensor and location data

ABSTRACT

Embodiments relate to using sensor data and location data from a device to generate augmented reality images. A mobile device pose can be determined (a geographic position, direction and a three dimensional orientation of the device) within a location. A type of destination in the location can be identified and multiple destinations can be identified, with the mobile device receiving queue information about the identified destinations from a server. A first image can be captured. Based on the queue information, one of the identified destinations can be selected. The geographic position of each identified destination can be identified, and these positions can be combined with the mobile device pose to generate a second image. Finally, an augmented reality image can be generated by combining the first image and the second image, the augmented reality image identifying the selected one destination.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of U.S. patent applicationSer. No. 13/289,292 (the '292 Application), currently pending, filed onNov. 4, 2011, which is continuation of U.S. patent application Ser. No.13/160,789, filed on Jun. 15, 2011, which claims the benefit andpriority of U.S. Provisional Application No. 61/355,000, filed Jun. 15,2010, each of these applications hereby being incorporated herein byreference in their entirety for all purposes.

FIELD

This disclosure relates in general to methods and systems forhighlighting physical features using a mobile device. More specifically,an augmented reality system is described that identifies features withina geographic location.

BACKGROUND

When a person is in a location having destinations of different types,it can often be difficult to evaluate criteria and select from multipledestinations. Once a destination is selected, it can be difficult tonavigate to the destination. Approaches to solving these problems do notprovide dynamic information about the destinations, don't have anintuitive way of moving to a selected destination, and aren't adapted tospecific types of locations where large groups of people congregate, andmany competing types of destinations exist.

SUMMARY

Some embodiments relate to using sensor data and location data from adevice to generate augmented reality images. A mobile device pose can bedetermined (a geographic position, direction and a three dimensionalorientation of the device) within a location. A type of destination inthe location can be identified and multiple destinations can beidentified, with the mobile device receiving queue information about theidentified destinations from a server. A first image can be captured.Based on the queue information, one of the identified destinations canbe selected. The geographic position of each identified destination canbe identified, and these positions can be combined with the mobiledevice pose to generate a second image. Finally, an augmented realityimage can be generated by combining the first image and the secondimage, the augmented reality image identifying the selected onedestination.

In some embodiments, a process for linking mobile devices at an eventusing groups based on seating and activity according to an embodiment;

In some embodiments, a system automatically groups attendees in a venuebased on affinity groups. Affinity groups can be based on a broad rangeof characteristics, including those retrieved from stored informationabout attendees and characteristics determined from behavior at theevent, e.g., movement, seating, etc.

In some embodiments, a process uses the displays of multiple mobiledevices to display a single image at an event, with each device becominga pixel of the displayed image. A still image (or a video frame) can bedivided up and sent to individual devices with displays (phones,tablets, smart watches) as well as displays in the venue. When enoughdevices are used, and the combination is viewed from a sufficientdistance, the still image (or moving video) is visible.

In some embodiments, images are automatically received from multiplemobile devices at an event according to different advantageous factors.Devices can be selected based on their current viewing position,proximity to an event of interest, capability of the device, speed ofconnection to the device, etc. Once selected, embodiments canautomatically trigger the capture of an image (or video) for collection,optional aggregation and display.

In some embodiments, captured sounds are automatically received frommultiple mobile devices at an event according to different advantageousfactors. Devices can be selected based on their current captureposition, proximity to an event of interest, capability of the device,speed of connection to the device, etc. Once selected, embodiments canautomatically trigger the capture of sound for collection, optionalaggregation and display.

In some embodiments, sounds are automatically outputted from multiplemobile devices at an event according to different advantageous factors.Devices can be selected based on their current output position,proximity to an audience of interest, capability of the device, speed ofconnection to the device, etc. Once selected, embodiments canautomatically trigger the output of sound.

An embodiment of a data presentation sculpture is described.

Further areas of applicability of the present disclosure will becomeapparent from the detailed description provided hereinafter. It shouldbe understood that the detailed description and specific examples, whileindicating various embodiments, are intended for purposes ofillustration only and are not intended to necessarily limit the scope ofthe disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is described in conjunction with the appendedfigures:

FIG. 1 depicts a block diagram of an embodiment of a resource managementsystem;

FIG. 2 depicts a block diagram of an embodiment of a resourceaccess-facilitating interaction system;

FIG. 3 shows an illustration of hardware and network connections of aresource access-facilitating interaction system according to anembodiment of the invention;

FIG. 4 shows an illustration of a communication exchange betweencomponents involved in a resource access-facilitating interaction systemaccording to an embodiment of the invention;

FIG. 5 illustrates example components of a device;

FIG. 6 illustrates example components of resource access coordinatormodule;

FIG. 7 illustrates a flowchart of an embodiment of a process forassigning access rights for resources;

FIG. 8A shows a block diagram of a site system according to anembodiment;

FIG. 8B shows another block diagram of a site system according to anembodiment;

FIG. 9 shows a block diagram of a mobile device according to anembodiment;

FIG. 10 shows a more detailed block diagram of a site system accordingto an embodiment;

FIG. 11 shows a block diagram of a wireless topology according to anembodiment;

FIG. 12 illustrates a flowchart of an embodiment of a process forestimating the location of the mobile device according to an embodiment;

FIG. 13 illustrates a flowchart of an embodiment of a process forlinking mobile devices at an event using groups based on seating andactivity according to an embodiment;

FIG. 14 shows a block diagram of a seating arrangement in a venueaccording to an embodiment;

FIG. 15 illustrates a timeline of event activities and attendee contactaccording to an embodiment;

FIG. 16 shows a block diagram of a system for automatically groupingattendees in a venue according to an embodiment;

FIG. 17 illustrates a flowchart of a process for using displays ofmultiple mobile devices to display a single image at an event accordingto an embodiment;

FIG. 18 illustrates a flowchart of another embodiment of a process forusing displays of multiple mobile devices to display a single image atan event;

FIG. 19 shows a diagram of using displays of multiple mobile devices todisplay a single image at an event according to an embodiment;

FIG. 20 illustrates a flowchart of an embodiment of a process fordisplaying images received from multiple mobile devices at an eventaccording to an embodiment;

FIG. 21 shows a diagram of a system for automatically triggering thecapture of media by multiple mobile devices at an event according to anembodiment;

FIG. 22 illustrates a timeline of an embodiment of a process forautomatically triggering the capture of media by multiple mobile devicesat an event according to an embodiment;

FIG. 23 illustrates a flowchart of an embodiment of a process for usingspeakers of multiple mobile devices to output sounds at an eventaccording to an embodiment;

FIG. 24 shows a diagram of a system for using speakers of multiplemobile devices to output sounds at an event according to an embodiment;

FIG. 25 illustrates a flowchart of an embodiment of a process forautomatically outputting media associated with an event to mobiledevices at the event according to an embodiment;

FIG. 26 shows a diagram of a system for automatically outputting mediaassociated with an event to mobile devices at the event according to anembodiment;

FIG. 27 shows a diagram of a data presentation sculpture according to anembodiment;

FIG. 28 shows an illustration of a mobile device having an augmentedreality interface according to an embodiment;

FIG. 29 shows a block diagram of an example system for the generation ofaugmented reality images according to an embodiment;

FIG. 30 illustrates an example captured image, according to anembodiment;

FIG. 31 illustrates an example overlay image, according to anembodiment;

FIG. 32 shows a diagram of an augmented reality image according to anembodiment; and

FIG. 33 illustrates a flowchart of an embodiment of a process forgenerating augmented reality images according to an embodiment.

In the appended figures, similar components and/or features can have thesame reference label. Further, various components of the same type canbe distinguished by following the reference label by a dash and a secondlabel that distinguishes among the similar components. If only the firstreference label is used in the specification, the description isapplicable to any one of the similar components having the same firstreference label irrespective of the second reference label.

DETAILED DESCRIPTION

The ensuing description provides preferred exemplary embodiment(s) only,and is not intended to limit the scope, applicability or configurationof the disclosure. Rather, the ensuing description of the preferredexemplary embodiment(s) will provide those skilled in the art with anenabling description for implementing a preferred exemplary embodiment.It is understood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope asset forth in the appended claims.

Referring first to FIG. 1, an embodiment of a resource management system100 is shown connected with various devices 110, 130, 140, 150, 160,170, and 180. The various devices 110, 130, 140, 150, 160, 170, and 180may each connect with the resource management system 100 in order toaccess an item or items of a resource controlled by the resourcemanagement system 100. Further, the resource management system 100controls whether or not the various devices 110, 130, 140, 150, 160,170, and 180 are granted access to resources. The various devices 110,130, 140, 150, 160, 170, and 180 may access the resource managementsystem 100 using mobile apps, web sites, call centers, venue boxoffices, application program interfaces (APIs), etc.

Mobile device 110 may connect to the resource management system 100 viaa web browser displayed on a display of the device, which facilitates awebpage for accessing the item or items of the resources. Desktopcomputer 130 may use a desktop computer to connect to the resourcemanagement system 100 via a web browser, which facilitates a webpage foraccessing the item or items of the resources. Tablet 140 may connectwith the resource management system 100 via a web browser displayed on adisplay of the tablet device, which facilitates a webpage for accessingthe item or items of the resources. The portable and/or handheldelectronic devices may include, for example, mobile phones, personaldigital assistant (PDA) devices. Laptop computer 170 may connect to theresource management system 100 via a browser displayed on a display ofthe laptop computer, which facilitates a webpage for accessing the itemor items of the resources. Lastly, intermediate system devices 180 mayinclude devices that enable brokers, group accessors, and/or wholesaleaccessors to access the items in a primary environment and controlaccess to rights to the items in a secondary environment. For example,the intermediate system devices 180 may be enabled to grant access tothe rights to the items accessed in the primary environment to thevarious devices 110, 130, 140, 150, 160, 170, and 180 in a secondaryenvironment. The intermediate system users 180 may connect to theresource management system 100 to access the resources using any of themeans discussed above.

In some cases, bot devices 150 connect to the resource management system100 to access the resources. Bot devices 150 include a server or serverfarm storing software, apps or scripts that automatically access allavailable items of resources without the need for user input by a human.Typically, bot devices 150 access or reserve available items in anabusive manner by accessing a large portion of available items andproviding rights to the reserved items to other users, therebypreventing fans (e.g., human users) from accessing the items directlyfrom the resource management system 100. That is, bot devices 150 do notaccess the resources in order to attend the event, whereas, fans andhuman users typically access the resources in order to attend the event.Further, the apps or scripts used by the bot devices 150 may transmitrequests to access the item or items via, for example, a webpage, atspeeds not possible by human users.

FIG. 2 depicts a block diagram of an embodiment of a resource managementsystem 100, according to an embodiment of the present disclosure. Mobiledevice 210 (which can be operated by a user 205) and an event-providerdevice 220 (which can be operated, controlled, or used by an eventprovider 215) can communicate with an access management system 285directly or via another system (e.g., via an intermediate system 250).Mobile device 210 may transmit data to access point 245, which isconnected to network 255, over communication channel 240 using antennae235. While FIG. 2 illustrates mobile device 210 communicating withaccess point 245 using a wireless connection (e.g., communicationchannel 240), in some embodiments, mobile device 210 may alsocommunicate with access point 245 using a wired connection (e.g., anEthernet connection). Mobile device 210 can also communicate with one ormore client devices, such as a client agent device 270 operated by aclient agent 275, a client register 260 or a client point device 265using a wired or wireless connection. In addition, using the accessmanagement system 285, an event provider 115 can identify an event, aparameter of attending the event, a date or dates of the event, alocation or locations of the event, etc. Each inter-system communicationcan occur over one or more networks 255 and can facilitate transmissionof a variety of types of data. It will be understood that, although onlyone of various systems, devices, entities and network are shown, theresource management system 100 can be extended to include multiple ofany given system(s), device(s), entity(ies), and/or networks. Sitesystem 280 is described further with the description of FIG. 10 below.

Access management system 285 can be configured to manage a dynamic setof access rights to one or more resources. More specifically, accessmanagement system 285 can track which resources are to be made availableto users, specifications of the resources and times at which they willbe available. Access management system 285 can also allocate accessrights for resources and facilitate transmissions of notifications ofthe available rights to a set of user devices. For example, accessmanagement system 285 can alert users of the availability via a website,app page or email. As another example, access management system cantransmit data about access rights and resources to one or moreintermediate systems 250, which can facilitate distribution ofaccess-right availability and processing of requests for such rights.

Notifications of available access rights can be accompanied by optionsto request that one or more access rights be assigned to a user.Therefore, user 205 can provide input to mobile device 210 via aninterface to request such assignment and provide other pertinentinformation. Intermediate system 250 and/or access management system 285can process the request to ensure that the requested access right(s)remain available and that all required information has been receivedand, in some instances, verified. Thereafter, access management system285 can assign one or more access rights to the user, e.g., matching theaccess rights requested by the user.

Assigning an access right can include, for example, associating anidentifier of the right with an identifier of a user, changing a statusof the right from available to assigned, facilitating a cease innotifications that the access right is available, generating anaccess-enabling code to use such that the corresponding access will bepermitted and/or generating a notification to be received at mobiledevice 210 confirming the assignment and/or including data required forcorresponding access to be permitted.

In some instances, a resource is at least partly controlled, by aclient. The resource may be accessed at a particular location orstructure, and a variety of client devices may be present at thelocation so as to facilitate usage of an access right. Exemplary clientdevices can include client agent device 270, which can be one operatedby a client agent 275 (e.g., a human client agent), a client register260 (e.g., which can operate independently of an agent and/or can beconnected to or include a device that, while in a locked mode, canimpede resource access, such as a turnstile) and client point device 265(e.g., which can operate independently of an agent and/or can bepositioned at or around the resource-associated location. For example,in some instances client agent device 270 can be operated by an agent ata location for a resource that is an event (“event resource”) takingplace at the location. In this example, client agent device 270 is usedby an agent that is manning an entrance to the location (e.g., which caninclude, for example, a location of a structure or a geographic region)or a part thereof; client register 260 can be or can be connected to aturnstile, gate or lockable door that is positioned along a perimeter orentrance to a resource-associated location or part thereof; and clientpoint device 265 can be an electronic device positioned at or within aresource-associated location.

In some instances, mobile device 210 performs particular functions upondetecting a client device and/or the contrary. For example, mobiledevice 210 may locally retrieve or request (e.g., from an externalsource) an access-enabling code. The access-enabling code can betransmitted to the client device or a remote server (e.g., a serverhosting access management system 285) for evaluation and/or can belocally evaluated. The evaluation can include, for example, confirmingthat the access-enabling code has a particular characteristic or format(e.g., generally or one characteristic corresponding to a particularresource or type of access), matches one in an access-enabling code datastore and/or has not been previously redeemed. A result of theevaluation can be locally displayed at an evaluating device, can controla device component (e.g., a physical access control module), and/or canbe transmitted to another device, such as mobile device 210.

In some instances, user 205 can use multiple mobile devices 210 toperform various operations (e.g., using one device to request an accessright and another to interact with client devices). Some instances ofmobile device 210, access management system 285, intermediate system250, client agent device 270, client register 260 and/or client pointdevice 265 can include a portable electronic device (e.g., a smartphone, tablet, laptop computer or smart wearable device) or anon-portable electronic device (e.g., one or more desktop computers,servers and/or processors).

In exemplary embodiments, access rights can be represented in datamaintained at a client device or at access management system 285. Forexample, a database or data store include a list of identifiers for eachuser or user device having an assigned access right for a resource orassociating an identifier for each user or user device with anidentifier of a particular access right. In some instances, indicia canbe transmitted to a user device that indicates that an access right isavailed. In various instances, it may be permitted or prohibited for theindicia to be transferred. The indicia may be provided as part of anelectronic or physical object (e.g., a right to access an event) orindependently. The indicia may include an access-enabling code.

In some instances, access management system 285 communicates with one ormore intermediate systems 250, each of which may be controlled by adifferent entity as compared to an entity controlling access managementsystem 285. For example, access management system 285 may assign accessrights to intermediate systems 250 (e.g., upon acceptance of terms).Intermediate system 250 can then collect data pertaining to the assignedaccess rights and/or a corresponding event, can format and/or edit thedata, generate a notification of availability of the access rights thatincludes the formatted and/or edited data and facilitate presentation ofthe notification at a mobile device 210. When intermediate system 250receives a communication from the mobile device 210 indicative of anaccess-right request, intermediate system 250 can facilitate assignment(or reassignment) of an access right to the user (e.g., by transmittingrelevant information to access management system 285 identifying theuser and/or user device and/or by transmitting relevant information tomobile device 210 pertaining to the access right).

A resource can include one managed or provided by a client, such as aperforming entity or an entity operating a venue. A mobile device 210can transmit data corresponding to the access right (e.g., anaccess-enabling code) to a client device upon, for example, detectingthe client device, detecting that a location of the mobile device 210 iswithin a prescribed geographical region, or detecting particular input.The receiving client device may include, for example, a client agentdevice 270 operated at an entrance of a defined geographical location ora client register 260 that includes or is attached to a lockingturnstile. The client device can then analyze the code to confirm itsvalidity and applicability for a particular resource and/or access type,and admittance to the event can be accordingly permitted. For example, aturnstile may change from a locked to an unlocked mode upon confirmationof the code's validity and applicability.

Each of the depicted devices and/or systems may include a software agentor application (“app”) that, when executed, performs one or more actionsas described herein. In some instances, a software agent or app on onedevice is, at least in part, complementary to a software agent or app onanother device (e.g., such that a software agent or app on mobile device210 is, at least in part, complementary to at least part of one onaccess management system 285 and/or a client device; and/or such that asoftware agent or app on intermediate system 250 is, at least in part,complementary to at least part of one on access management system 285).

In some instances, a network in the one or more networks 255 can includean open network, such as the Internet, personal area network, local areanetwork (LAN), campus area network (CAN), metropolitan area network(MAN), wide area network (WAN), wireless local area network (WLAN), aprivate network, such as an intranet, extranet, or other backbone. Insome instances, a network in the one or more networks 255 includes ashort-range communication channel, such as Bluetooth or Bluetooth LowEnergy channel. Communicating using a short-range communication such asBLE channel can provide advantages such as consuming less power, beingable to communicate across moderate distances, being able to detectlevels of proximity, achieving high-level security based on encryptionand short ranges, and not requiring pairing for inter-devicecommunications.

In one embodiment, communications between two or more systems and/ordevices can be achieved by a secure communications protocol, such assecure sockets layer (SSL), transport layer security (TLS). In addition,data and/or transactional details may be encrypted based on anyconvenient, known, or to be developed manner, such as, but not limitedto, DES, Triple DES, RSA, Blowfish, Advanced Encryption Standard (AES),CAST-128, CAST-256, Decorrelated Fast Cipher (DFC), Tiny EncryptionAlgorithm (TEA), eXtended TEA (XTEA), Corrected Block TEA (XXTEA),and/or RC5, etc.

It will be appreciated that, while a variety of devices and systems areshown in FIG. 1, in some instances, resource management system 100 caninclude fewer devices and/or systems. Further, some systems and/ordevices can be combined. For example, a client agent device 270 may alsoserve as an access management system 285 or intermediate system 250 soas to as to facilitate assignment of access rights.

As described in further detail herein, an interaction between mobiledevice 210 and a client device (e.g., client agent device 270, clientregister 260 or client point device 265) can facilitate, for example,verification that user 205 has a valid and applicable access right,obtaining an assignment of an access right, and/or obtaining anassignment of an upgraded access right.

In addition, mobile device 210-2, which is operated by user 225-2, mayinclude a user device that is located at a stadium or concert hallduring an event. Mobile device 210-2 may directly interact with a clientdevice (e.g., client agent device 270, client register 260 or clientpoint device 265), which is also located at the stadium or concert hallduring the event. As such, the access management system 285 may beupdated or accessed by mobile device 210-2 via the client agent device270. For example, mobile device 210-2 may communicate with the clientagent device 270 over a short-range communication channel 290, such asBluetooth or Bluetooth Low Energy channel, Near Field Communication(NFC), Wi-Fi, RFID, Zigbee, ANT, etc. Communicating using a short-rangecommunication such as BLE channel can provide advantages such asconsuming less power, being able to communicate across moderatedistances, being able to detect levels of proximity, achievinghigh-level security based on encryption and short ranges, and notrequiring pairing for inter-device communications. After the short-rangecommunication link 290 is established, mobile device 210-2 maycommunicate with the access management system 285 and access the item oritems of resources. That is, while mobile device B is configured tocommunicate over network 255, mobile device 210-2 may communicate withthe access management system 285 via the client agent device 270,instead of the network 255.

It will be appreciated that various parts of system 200 can begeographically separated. It will further be appreciated that system 200can include a different number of various components rather than anumber depicted in FIG. 2. For example, two or more of access assignmentsystems 285; one or more site systems 280; and intermediate system 250may be located in different geographic locations (e.g., differentcities, states or countries).

FIG. 3 shows an illustration of hardware and network connections of aresource access-facilitating interaction system 300 according to anembodiment of the invention. Each of various user devices 310-1, 310-2,310-3, 310-4 and 310-5 can connect, via one or more inter-networkconnection components (e.g., a router 312) and one or more networks 370to a primary assignment management system 314 or a secondary assignmentmanagement system 316-1, 316-2 or 316-3.

Primary assignment management system 314 can be configured to coordinateand/or control initial assignment of access rights. Secondary assignmentmanagement system 316 can be configured to coordinate and/or controlreassignment and/or transfer of access rights (e.g., from one user oruser device to another or from an intermediate agent to a user or userdevice). Such transfer may occur as a result of a sale or fee payment.Secondary assignment management system 316 may also manage transferoffers (e.g., to allow a first user to identify a price at which atransfer request would be granted and to detect if a valid request isreceived). It will be appreciated that, although primary assignmentmanagement system 314 is shown to be separate from each secondaryassignment management system 316, in some instances, an assignmentmanagement system may relate to both a primary and secondary channel,and a single data store or a localized cluster of data stores mayinclude data from both channels.

Each of primary access assignment system 314 and secondary accessassignment system 316 can include a web server 318 that processes andresponds to HTTP requests. Web server 318 can retrieve and deliverweb-page data to a user device 310 that, for example, identify aresource, identify a characteristic of each of one or more access rightsfor the resource, include an invitation to request assignment of anaccess right, facilitate establishment or updating of an account, and/oridentify characteristics of one or more assigned access rights. Webserver 318 can be configured to support server-side scripting and/orreceive data from user devices 310, such as data from forms or fileuploads.

In some instances, a web server 318 can be configured to communicatedata about a resource and an indication that access rights for theresource are available. Web server 318 can receive a requestcommunication from a user device 310 that corresponds to a request forinformation about access rights. The request can include one or moreconstraints, which can correspond to (for example) values (e.g., to bematched or to define a range) of particular fields.

A management server 322 can interact with web server 318 to provideindications as to which access rights' are available for assignment,characteristics of access rights and/or what data is needed to assign anaccess right. When requisite information is received (e.g., about a userand/or user device, identifying a final request for one or more accessrights, including payment information, and so on), management server 322can coordinate an assignment of the one or more access rights. Thecoordination can include updating an access-right data store to change astatus of the one or more access rights (e.g., to assigned); toassociate each of the one or more access rights with a user and/or userdevice; to generate or identify one or more access-enabling codes forthe one or more access rights; and/or to facilitate transmissionreflecting the assignment (e.g., and including the one or moreaccess-enabling codes) to a user device.

Management server 322 can query, update and manage an access-right datastore to identify access rights' availability and/or characteristicand/or to reflect a new assignment. The data store can include oneassociated with the particular assignment system. In some instances, thedata store includes incomplete data about access rights for a resource.For example, a data store 324 at and/or used by a secondary accessassignment system 316 may include data about an incomplete subset ofaccess rights that have been allocated for a particular resource. Toillustrate, a client agent may have indicated that an independentintermediary system can (exclusively or non-exclusively) coordinateassignment of a portion of access rights for a resource but not theremainder. A data store 324 may then, for example, selectively includeinformation (e.g., characteristics, statuses and/or assignmentassociations) for access rights in the portion.

Data store 324 or 326 associated with a particular primary or secondaryaccess assignment system can include assignment data for a set of accessrights that are configured to be set by the particular primary orsecondary access assignment system or by another system. For example, arule can indicate that a given access right is to have an availablestatus until a first of a plurality of access assignment systems assignsthe access right. Accordingly, access assignment systems would then needto communicate to alert each other of assignments.

In one instance, management server 322 (or another server in an accessassignment system) sends a communication to a central data managementserver farm 328 reflecting one or more recent assignments. Thecommunication may include an identification of one or more accessrights, an indication that the access right(s) have been assigned, anidentification of a user and/or user device associated with theassignment and/or one or more access-enabling codes generated oridentified to be associated with the assignment. The communication canbe sent, for example, upon assigning the access right(s), as a precursorto assigning the access right(s) (e.g., to confirm availability and/orrequest assignment authorization), at defined times or time intervalsand/or in response to an assignment-update request received from datamanagement server farm 328.

Data management server farm 328 can then update a central data store toreflect the data from the communication. The central data store can bepart of, for example, a network-attached storage 332 and/or astorage-area network 334.

In some instances, a data store 324 or 326 can include a cache, thatincludes data stored based on previous communications with datamanagement server farm 328. For example, data management server farm 328may periodically transmit statuses of a set of access rights (e.g.,those initially configured to be assignable by an access assignmentsystem) or an updated status (e.g., indicating an assignment) of one ormore access rights. As another example, data management server farm 328may transmit statuses upon receiving a request from an access assignmentsystem for statuses and/or authorization to assign one or more accessrights.

An access assignment system may receive statuses less frequently or attimes unaligned with requests received from user devices requestinginformation about access rights and/or assignments. Rather than initiatea central data store query responsive to each user-device request, amanagement server 322 can rely on cached data (e.g., locally cacheddata) to identify availability of one or more access rights, as reflectin webpage data and/or communications responsive to requestcommunications for access-right information. After requisite informationhas been obtained, management server 322 can then communicate with datamanagement server farm 328 to ensure that one or more particular accessrights have remained available for assignment.

In some instances, one or more of primary access assignment system 314and/or a secondary access assignment system 314 need not include a localor system-inclusive data store for tracking access-right statuses,assignments and/or characteristics. Instead, the access assignmentsystem may communicate with a remote and/or central data store (e.g.,network-attached storage 332 or storage-area network 334).

Access management system 285 can include a primary access assignmentsystem 314 and/or a secondary access assignment system 314; datamanagement server farm 328; and/or a central data store (e.g.,network-attached storage 332 or storage-area network 334). Each of oneor more intermediate systems 250 can include a primary access assignmentsystem 314 and/or a secondary access assignment system 314.

Data management server farm 328 may periodically and/or routinely assessa connection with an access assignment system 314. For example, a testcommunication can be sent that is indicative of a request to respond(e.g., with particular data or generally). If a response communicationis not received, if a response communication is not received within adefined time period and/or if a response communication includesparticular data (e.g., reflecting poor data integrity, network speed,processing speed, etc.), data management server farm 328 may reconfigureaccess rights and/or permissions and/or may transmit anothercommunication indicating that assignment rights of the access assignmentsystem are limited (e.g., to prevent the system from assigning accessrights).

It will be appreciated that various parts of system 300 can begeographically separated. For example, two or more of primary accessassignment system 314; one or more of secondary access assignmentsystems 314; and data management server farm 328 may be located indifferent geographic locations (e.g., different cities, states orcountries).

It will further be appreciated that system 300 can include a differentnumber of various components rather than a number depicted in FIG. 3.For example, system 300 can include multiple data management serverfarms 328, central data stores and/or primary access assignment systems314 (e.g., which can be geographically separated, such as being locatedin different cities, states or countries). In some instances, processingmay be split (e.g., according to a load-balancing technique) acrossmultiple data management server farms 328 and/or across multiple accessassignment systems 314. Meanwhile, the farms and/or systems can beconfigured to accept an increased or full load should another farmand/or system be unavailable (e.g., due to maintenance). Data stored ina central data store may also be replicated in geographically separateddata stores.

FIG. 4 shows an illustration of a communication exchange betweencomponents involved in a resource access-facilitating interaction system400 according to an embodiment of the invention. A user device 410 cansend one or more HTTP requests to a web-server system 418, andweb-server system 418 can respond with one or more HTTP responses thatinclude webpage data. User device 410, in some embodiments, can be amobile device similar to mobile devices 110 and 210 from FIGS. 1 and 2respectively. The webpage data can include, for example, informationabout one or more resources, characteristics of a set of access rightsfor each of the one or more resources, availability of one or moreaccess rights, an invitation to request an assignment of one or moreaccess rights and/or indications as to what information is required foran access-right assignment. HTTP requests can include assignment-requestdata (e.g., a resource identification, requisite information, and/or anidentification of an access-right constraint or access right).

Web-server system 418 can include one or more web processors (e.g.,included in one or more server farms, which may be geographicallyseparated) to, for example, map a path component of a URL to web data(e.g., stored in a local file system or generated by a program);

retrieve the web data; and/or generate a response communicationincluding the web data. Web processor can further parse communication toidentify input-corresponding data in HTTP requests, such as field valuesrequired for an access-right assignment.

Web-server system 418 can also include a load balancer to distributeprocessing tasks across multiple web processors. For example, HTTPrequests can be distributed to different web processors. Load-balancingtechniques can be configured so as, for example, to distributeprocessing across servers or server farms, decrease a number of hopsbetween a web server and user device, decrease a geographical locationbetween a user device and web server, etc.

Web-server system 418 can further include a RAID component, such as aRAID controller or card. A RAID component can be configured, forexample, to stripe data across multiple drives, distribute parity acrossdrives and/or mirror data across multiple drives. The RAID component canbe configured to improve reliability and increase request-processingspeeds.

Web-server system 418 can include one or more distributed,non-distributed, virtual, non-virtual, local and/or remote data stores.The data stores can include web data, scripts and/or content object(e.g., to be presented as part or web data).

Some HTTP requests include requests for identifications of access-rightcharacteristics and/or availability. To provide web data reflecting suchinformation, web-server system 418 can request the information fromanother server, such as an SQL system 441 (e.g., which may include oneor more servers or one or more server farms).

SQL system 441 can include one or more SQL processors (e.g., included inone or more server farms, which may be geographically separated). SQLprocessors can be configured to query, update and otherwise use one ormore relational data stores. SQL processors can be configured to execute(and, in some instances, generate) code (e.g., SQL code) to query arelational data store.

SQL system 441 can include a database engine, that includes a relationalengine, OLE database and storage engine. A relational engine canprocess, parse, compile, and/or optimize a query and/or makequery-associated calls. The relational engine can identify an OLE DB rowset that identifies the row with columns matching search criteria and/ora ranking value. A storage engine can manage data access and use therowset (e.g., to access tables and indices) to retrieve query-responsivedata from one or more relational databases.

SQL system 441 can include one or more distributed, non-distributed,virtual, non-virtual, local and/or remote relational data stores. Therelational databases can include linked data structures identifying, forexample, resource information, access-right identifications andcharacteristics, access-right statuses and/or assignments, and/or userand/or user account data. Thus, for example, use of the relationalstructures may facilitate identifying, for a particular user, acharacteristic of an assigned access right and information about aresource associated with the access right.

One or more data structures in a relational data structure may reflectwhether particular access rights have been assigned or remain available.This data may be based on data received from a catalog system 442 thatmonitors and tracks statuses of resource access rights. Catalog system442 can include one or more catalog processors (e.g., included in one ormore server farms, which may be geographically separated). Catalogprocessors can be configured to generate status-update requestcommunications to be sent to one or more access assignment systemsand/or intermediate systems and/or to receive status-updatecommunications from one or more access assignment systems and/orintermediate systems. A status-update communication can, for example,identify an access right and/or resource and indicate an assignment ofthe access right. For example, a status-update communication canindicate that a particular access right has been assigned and is thus nolonger available. In some instances, a status-update communicationidentifies assignment details, such as a user, account and/or userdevice associated with an access-right assignment; a time that theassignment was made; and/or a price associated with the assignment.

In some instances, a status update is less explicit. For example, acommunication may identify an access right and/or resource and request afinal authorization of an assignment of the access right. Catalog system442 can then verify that the access right is available for assignment(e.g., and that a request-associated system or entity is authorized tocoordinate the assignment) and can transmit an affirmative response.Such a communication exchange can indicate (in some instances) that theaccess right is assigned and unavailable for other assignment.

In some instances, catalog system 442 can also be integrated with anon-intermediate access assignment system, such that it can directlydetect assignments. For example, an integrated access assignment systemcan coordinate a message exchange with a user device, can query acatalog data store to identify available access rights and canfacilitate or trigger a status-change of an access right to reflect anassignment (e.g., upon having received all required information.

Whether a result of a direct assignment detection or a status updatefrom an intermediate system, a database engine of catalog system 442 canmanage one or more data stores so as to indicate a current status ofeach of a set of access rights for a resource. The one or more datastores may further identify any assignment constraints. For example,particular access rights may be earmarked so as to only allow one ormore particular intermediate systems to trigger a change to the accessrights' status and/or to assign the access rights.

The database engine can include a digital asset management (DAM) engineto receive, transform (e.g., annotate, reformat, introduce a schema,etc.) status-update communications, and identify other data (e.g., anidentifier of an assigning system and/or a time at which a communicationwas received) to associate with a status update (e.g., an assignment).Therefore, the DAM engine can be configured to prepare storage-updatetasks so as to cause a maintained data store to reflect a recent datachange.

Further, the DAM engine can facilitate handling of data-store queries.For example, a status-request communication or authorization-requestcommunication can be processed to identify variables and/or indices touse to query a data store. A query can then be generated and/or directedto a data store based on the processing. The DAM engine can relay (e.g.,and, potentially, perform intermediate processing to) a query result toa request-associate system.

The database engine can also include a conflict engine, which can beconfigured to access and implement rules indicating how conflicts are tobe handled. For example, catalog system 442 may receive multiplerequests within a time period requesting an assignment authorization (ora hold) for a particular access right. A rule may indicate that a firstrequest is to receive priority, that a request associated with a morehighly prioritized requesting system (e.g., intermediate system) is tobe prioritized, that a request associated with a relatively high (orlow) quantity of access rights identified in the request for potentialassignment are to be prioritized, etc.

The database engine can further include a storage engine configured tomanage data access and/or data updates (e.g., modifying existing data oradding new data). The data managed by and/or accessible to the storageengine can be included in one or more data stores. The data stores caninclude, for example, distributed, non-distributed, virtual,non-virtual, local and/or remote data stores. The data stores caninclude, for example, a relational, non-relational, object, non-object,document and/or non-document data store. Part or all of a data store caninclude a shadow data store that shadows data from another data store.Part or all of a data store can include an authoritative data store thatis (e.g., directly and/or immediately) updated with access-rightassignment changes (e.g., such that a primary or secondary accessassignment system updates the data store as part of an access-rightassignment process, rather than sending a post-hoc status-updatecommunication reflecting the assignment). In some instances, a datastore an authoritative data store identifies a status for each of a set(e.g., or all) of access rights for a given resource. Should there beany inconsistency between an authoritative data store and another datastore (e.g., at an intermediate system), system 400 can be configuredsuch that the authoritative data store is controlling.

System 400 can further include a replication system 443. Replicationsystem 443 can include one or more replication processors configured toidentify new or modified data, to identify one or more data storesand/or location at which to store the new or modified data and/or tocoordinate replication of the data. In some instances, one or more ofthese identifications and/or coordination can be performed using areplication rule. For example, a replication rule may indicate thatreplication is to be performed in a manner biased towards storingreplicated data at a data store geographically separated from anotherdata store storing the data.

A data duplicator can be configured to read stored data and generate oneor more write commands so as to store the data at a different datastore. A controller can manage transmitting write commands appropriatelyso as to facilitate storing replicated data at identified data stores.Further, a controller can manage data stores, such as a distributedmemory or distributed shared memory, to ensure that a currently activeset of data stores includes a target number of replications of data.

Accordingly, web-server system 418 can interact with user device 410 toidentify available access rights and to collect information needed toassign an access right. Web-server system 418 can interact with SQLsystem 441 so as to retrieve data about particular resources and/oraccess rights so as to configure web data (e.g., via dynamic webpages orscripts) to reflect accurate or semi-accurate information and/orstatuses. SQL system 441 can use relational data stores to quicklyprovide such data. Meanwhile, catalog system 442 may manage one or morenon-relational and/or more comprehensive data stores may be tasked withmore reliably and quickly tracking access-right statuses andassignments. The tracking may include receiving status updates (e.g.,via a push or pull protocol) from one or more intermediate systemsand/or by detecting assignment updates from non-intermediate systems,such as an integrated access assignment system and/or SQL system 441.Catalog system 442 may provide condensed status updates (e.g.,reflecting a binary indication as to whether an access right isavailable) to SQL system 441 periodically, at triggered times and/or inresponse to a request from the SQL system. A replication system 443 canfurther ensure that data is replicated at multiple data stores, so as toimprove a reliability and speed of system 400.

It will be appreciated that various parts of system 400 can begeographically separated. For example, each of user device 410,intermediate system 430, web-server system 418, SQL system 441, catalogsystem 442 and replication 443 may be located in different geographiclocations (e.g., different cities, states or countries).

FIG. 5 illustrates example components of a device 500, such as a clientdevice (e.g., client agent device 270, client register 260 and/or clientpoint device 265), an intermediate system (e.g., intermediate system(s)250) and/or an access management system (e.g., access management system285) according to an embodiment of the invention.

The components can include one or more modules that can be installed ondevice 500. Modules can include some or all of the following: a networkinterface module 502 (which can operate in a link layer of a protocolstack), a message processor module 504 (which can operate in an IP layerof a protocol stack), a communications manager module 506 (which canoperate in a transport layer of a protocol stack), a communicationsconfigure module 508 (which can operate in a transport and/or IP layerin a protocol stack), a communications rules provider module 510 (whichcan operate in a transport and/or IP layer in a protocol stack),application modules 512 (which can operate in an application layer of aprotocol stack), a physical access control module 532 and one or moreenvironmental sensors 534.

Network interface module 502 receives and transmits messages via one ormore hardware components that provide a link-layer interconnect. Thehardware component(s) can include, for example, RF antenna 503 or a port(e.g., Ethernet port) and supporting circuitry. In some embodiments,network interface module 502 can be configured to support wirelesscommunication, e.g., using Wi Fi (IEEE 802.11 family standards),Bluetooth® (a family of standards promulgated by Bluetooth SIG, Inc.),BLE, or near-field communication (implementing the ISO/IEC 18092standards or the like).

RF antenna 503 can be configured to convert electric signals into radioand/or magnetic signals (e.g., to radio waves) to transmit to anotherdevice and/or to receive radio and/or magnetic signals and convert themto electric signals. RF antenna 503 can be tuned to operate within aparticular frequency band. In some instances, a device includes multipleantennas, and the antennas can be, for example, physically separated. Insome instances, antennas differ with respect to radiation patterns,polarizations, take-off angle gain and/or tuning bands. RF interfacemodule 502 can include one or more phase shifters, filters, attenuators,amplifiers, switches and/or other components to demodulate receivedsignals, coordinate signal transmission and/or facilitate high-qualitysignal transmission and receipt.

In some instances, network interface module 502 includes a virtualnetwork interface, so as to enable the device to utilize an intermediatedevice for signal transmission or reception. For example, networkinterface module 502 can include VPN software.

Network interface module 502 and one or more antennas 503 can beconfigured to transmit and receive signals over one or more connectiontypes. For example, network interface module 502 and one or moreantennas 503 can be configured to transmit and receive WiFi signals,cellular signals, Bluetooth signals, Bluetooth Low Energy (BLE) signals,Zigbee signals, or Near-Field Communication (NFC) signals.

Message processor module 504 can coordinate communication with otherelectronic devices or systems, such as one or more servers or a userdevice. In one instance, message processor module 504 is able tocommunicate using a plurality of protocols (e.g., any known, futureand/or convenient protocol such as, but not limited to, XML, SMS, MMS,and/or email, etc.). Message processor module 504 may further optionallyserialize incoming and/or outgoing messages and facilitate queuing ofincoming and outgoing message traffic.

Message processor module 504 can perform functions of an IP layer in anetwork protocol stack. For example, in some instances, messageprocessor module 504 can format data packets or segments, combine datapacket fragments, fragment data packets and/or identify destinationapplications and/or device addresses. For example, message processormodule 504 can defragment and analyze an incoming message to determinewhether it is to be forwarded to another device and, if so, can addressand fragment the message before sending it to the network interfacemodule 502 to be transmitted. As another example, message processormodule 504 can defragment and analyze an incoming message to identify adestination application that is to receive the message and can thendirect the message (e.g., via a transport layer) to the application.

Communications manager module 506 can implement transport-layerfunctions. For example, communications manager module 506 can identify atransport protocol for an outgoing message (e.g., transmission controlprotocol (TCP) or user diagram protocol (UDP)) and appropriatelyencapsulate the message into transport protocol data units. Messageprocessor module 504 can initiate establishment of connections betweendevices, monitor transmissions failures, control data transmission ratesand monitoring transmission quality. As another example, communicationsmanager module 506 can read a header of an incoming message to identifyan application layer protocol to receive the message's data. The datacan be separated from the header and sent to the appropriateapplication. Message processor module 504 can also monitor the qualityof incoming messages and/or detect out of order incoming packets.

In some instances, characteristics of message-receipt ormessage-transmission quality can be used to identify a health status ofan established communications link. In some instances, communicationsmanager module 506 can be configured to detect signals indicating thehealth status of an established communications link (e.g., a periodicsignal from the other device system, which if received without dropouts,indicates a healthy link).

In some instances, a communication configurer module 508 is provided totrack attributes of another system so as to facilitate establishment ofa communication session. In one embodiment, communication configurermodule 508 further ensures that inter-device communications areconducted in accordance with the identified communication attributesand/or rules. Communication configurer module 508 can maintain anupdated record of the communication attributes of one or more devices orsystems. In one embodiment, communications configurer module 508 ensuresthat communications manager module 206 can deliver the payload providedby message processor module 504 to the destination (e.g., by ensuringthat the correct protocol corresponding to the client system is used).

A communications rules provider module 510 can implement one or morecommunication rules that relate to details of signal transmissions orreceipt. For example, a rule may specify or constrain a protocol to beused, a transmission time, a type of link or connection to be used, adestination device, and/or a number of destination devices. A rule maybe generally applicable or conditionally applicable (e.g., only applyingfor messages corresponding to a particular app, during a particular timeof day, while a device is in a particular geographical region, when ausage of a local device resource exceeds a threshold, etc.). Forexample, a rule can identify a technique for selecting between a set ofpotential destination devices based on attributes of the set ofpotential destination devices as tracked by communication configuremodule 316. To illustrate, a device having a short response latency maybe selected as a destination device. As another example, communicationsrules provider 510 can maintain associations between various devices orsystems and resources. Thus, messages corresponding to particularresources can be selectively transmitted to destinations having accessto such resources.

A variety of application modules 512 can be configured to initiatemessage transmission, process incoming transmissions, facilitateselective granting of resource access, facilitate processing of requestsfor resource access, and/or performing other functions. In the instancedepicted in FIG. 5, application modules 512 include an auto-updatermodule 514, a resource access coordinator module 516, and/or a codeverification module 518.

Auto-updater module 514 automatically updates stored data and/or agentsoftware based on recent changes to resource utilization, availabilityor schedules and/or updates to software or protocols. Such updates canbe pushed from another device (e.g., upon detecting a change in aresource availability or access permit) or can be received in responseto a request sent by device 500. For example, device 500 can transmit asignal to another device that identifies a particular resource, and aresponsive signal can identify availabilities of access to the resource(e.g., available seat reservations for a sporting event or concert). Asanother example, device 500 can transmit a signal that includes anaccess access-enabling code, and a responsive signal can indicatewhether the code is applicable for access of a particular resourceand/or is valid.

In some instances, auto-updater module 514 is configured to enable theagent software to understand new, messages, commands, and/or protocols,based on a system configuration/change initiated on another device.Auto-updater module 514 may also install new or updated software toprovide support and/or enhancements, based on a system configurationchange detected on device 500. System configuration changes that wouldnecessitate changes to the agent software can include, but are notlimited to, a software/hardware upgrade, a security upgrade, a routerconfiguration change, a change in security settings, etc. For example,if auto-updater module 514 determines that a communication link withanother device has been lost for a pre-determined amount of time,auto-updater module 514 can obtain system configuration information tohelp re-establish the communication link. Such information may includenew settings/configurations on one or more hardware devices or new orupgraded software on or connected to device 500. Thus, auto-updatermodule 514 can detect or be informed by other software when there is anew version of agent software with additional functionality and/ordeficiency/bug corrections or when there is a change with respect to thesoftware, hardware, communications channel, etc.), and perform updatesaccordingly.

Based on the newly obtained system configuration for device 500,auto-updater module 514 can cause a new communication link to bere-established with another device. In one embodiment, uponestablishment of the communication link, system configurationinformation about device 500 can also be provided to another device tofacilitate the connection to or downloading of software to device 500.

In one embodiment, when a poor health signal is detected by anotherdevice (e.g., when the health signal is only sporadically received butthe communication link is not necessarily lost), the other device cansend a command to auto-updater module 514 to instruct auto-updatermodule 514 to obtain system configuration information about device 500.The updated system configuration information may be used in an attemptto revive the unhealthy communications link (e.g., by resending aresource request). For example, code can utilize appropriate systemcalls for the operating system to fix or reestablish communications. Byway of example and not limitation, model and driver information isoptionally obtained for routers in the system in order querying them. Byway of further example, if the code determines that a new brand ofrouter has been installed, it can adapt to that change, or to the changein network configuration, or other changes.

Instead or in addition, the host server (e.g., via communicationsmanager 506) can send specific instructions to auto-updater module 514to specify tests or checks to be performed on device 500 to determinethe changes to the system configurations (e.g., by automaticallyperforming or requesting an inventory check of system hardware and/orsoftware). For example, the components involved in the chain of hopsthrough a network can be queried and analyzed. Thus, for example, if anew ISP (Internet service provider) is being used and the managementsystem traffic is being filtered, or a new router was installed and thesoftware needs to change its configuration, or if someone made a changeto the operating system that affects port the management system is usingto communicate, the management system (or operator) can communicate withthe ISP, change it back, or choose from a new available port,respectively.

The specific tests may be necessary to help establish the communicationlink, if, for example, the automatic tests fail to provide sufficientinformation for the communication link to be re-established, ifadditional information is needed about a particular configurationchange, and/or if the client system is not initially supported by theauto-updater module 514, etc.

Auto-updater module 514 can also receive signals identifying updatespertaining to current or future availability of resources and/or accesspermits. Based on the signals, auto-updater module 514 can modify, addto or delete stored data pertaining to resource availabilities, resourceschedules and/or valid access permits. For example, upon receiving anupdate signal, auto-updater 514 can modify data stored in one or moredata stores 522, such as an account data store 524, resourcespecification data store 526, resource status data store 528 and/oraccess-enabling code data store 530.

Account data store 524 can store data for entities, such asadministrators, intermediate-system agents and/or users. The accountdata can include login information (e.g., username and password),identifying information (e.g., name, residential address, phone number,email address, age and/or gender), professional information (e.g.,occupation, affiliation and/or professional position), preferences(e.g., regarding event types, performers, seating areas, and/or resourcetypes), purchase data (e.g., reflecting dates, prices and/or items ofpast purchases) and/or payment data (e.g., credit card number andexpiration date or payment account information). The account data canalso or alternatively include technical data, such a particular entitycan be associated with one or more device types, IP addresses, browseridentifier and/or operating system identifier).

Resource specification data store 526 can store specification datacharacterizing each of one or more resources. For example, specificationdata for a resource can include a processing power, available memory,operating system, compatibility, device type, processor usage, powerstatus, device model, number of processor cores, types of memories, dateand time of availability, a performing entity, a venue of the eventand/or a set of seats (e.g., a chart or list). Specification data canfurther identify, for example, a cost for each of one or more accessrights.

Resource status data store 528 can store status data reflecting whichresources are available (or unavailable), thereby indicating whichresources have one or more open assignments. In some instances, thestatus data can include schedule information about when a resource isavailable. Status data can include information identifying an entity whorequested, reserved or was assigned a resource. In some instances,status information can indicate that a resource is being held orreserved and may identify an entity associated with the hold or reserveand/or a time at which the hold or reservation will be released.

Access-enabling code data store 530 can store access-enabling code datathat includes one or more codes and/or other information that can beused to indicate that an entity is authorized to use, have or receive aresource. An access-enabling code can include, for example, a numericstring, an alphanumeric string, a text string, a 1-dimensional code, a2-dimensional code, a barcode, a quick response (QR) code, an image, astatic code and/or a temporally dynamic code. An access-enabling codecan be, for example, unique across all instances, resource types and/orentities. For example, access-enabling codes provided in association fortickets to a particular event can be unique relative to each other. Insome instances, at least part of a code identifies a resource orspecification of a resource. For example, for a ticket to a concert,various portions of a code may reflect: a performing entity, resourcelocation, date, section and access-permitted location identifier.

One or more of data stores 524, 526, 528, and 530 can be a relationaldata store, such that elements in one data store can be referencedwithin another data store. For example, resource status data store 528can associate an identifier of a particular ticket with an identifier ofa particular entity. Additional information about the entity can then beretrieved by looking up the entity identifier in account data store 524.

Updates to data stores 524, 526, 528, and 530 facilitated and/orinitiated by auto-updater module 514 can improve cross-device dataconsistency. Resource access coordinator module 516 can coordinateresource access by, for example, generating and distributingidentifications of resource availabilities; processing requests forresource access; handling competing requests for resource access; and/orreceiving and responding to resource-offering objectives.

FIG. 6 illustrates example components of resource access coordinatormodule 516 that may operate, at least in part, at an access managementsystem (e.g., access management system) according to an embodiment ofthe invention. A resource specification engine 602 can identify one ormore available resources. For example, resource specification engine 602can detect input that identifies a current or future availability of anew resource.

Resource specification engine 602 can identify one or morespecifications of each of one or more resources. A specification caninclude an availability time period. For example, resource specificationengine 602 can determine that a resource is available, for example, at aparticular date and time (e.g., as identified based on input), for atime period (e.g., a start to end time), as identified in the input,and/or from a time of initial identification until another inputindicating that the resource is unavailable is detected. A specificationcan also or alternatively include a location (e.g., a geographiclocation and/or venue) of the resource. A specification can also oralternatively include one or more parties associated with the resource(e.g., performing acts or teams). Resource specification engine 602 canstore the specifications in association with an identifier of theresource in resource specifications data store 526.

A resource-access allocation engine 604 can allocate access rights forindividual resources. An access right can serve to provide an associatedentity with the right or a priority to access a resource. Because (forexample) association of an access right with an entity can, in someinstances, be conditioned on fee payment or authorization thereof, anallocated access right can be initially unassociated with particularentities (e.g., users). For example, an allocated right can correspondto one or more access characteristics, such as an processor identifier,a usage time, a memory allocation, a geographic location (e.g., sectionor seat identifier), and/or a fee. For an allocated access right,resource-access allocation engine 604 can store an identifier of theright in resource statuses data store 528 in association with anidentifier for the resource and an indication that it has not yet beenassigned to a particular entity.

A communication engine 606 can facilitate communicating the availabilityof the resource access rights to users. In some instances, a publisherengine 608 generates a presentation that identifies a resource andindicates that access rights are available. Initially or in response touser interaction with the presentation, the presentation can identifyaccess characteristics about available access rights. The presentationcan include, for example, a chart that identifies available accessrights for an event and corresponding fees. Publisher engine 608 candistribute the presentation via, for example, a website, app page, emailand/or message. The presentation can be further configured to enable auser to request assignments of one or more access rights.

In some instances, an intermediate system coordination engine 610 canfacilitate transmission of information about resource availability(e.g., resource specifications and characteristics of resource-accessrights) to one or more intermediate systems (e.g., by generating one ormore messages that include such information and/or facilitatingpublishing such information via a website or app page). Each of the oneor more intermediate systems can publish information about the resourceand accept requests for resource access. In some instances, intermediatesystem coordination engine 610 identifies different access rights asbeing available to individual intermediate systems to coordinateassignment. For example, access rights for seats in Section 1 may beprovided for a first intermediate system to assign, and access rightsfor seats in Section 2 may be provided to a second intermediate systemto assign.

In some instances, overlapping access rights are made available tomultiple intermediate systems to coordinate assignments. For example,some or all of a first set of resource rights (e.g., corresponding to asection) may be provided to first and second intermediate systems. Insuch instances, intermediate system coordination engine 610 can respondto a communication from a first intermediate system indicating that arequest has been received (e.g., and processed) for an access right inthe set) by sending a notification to one or more other intermediatesystems that indicates that the access right is to be at leasttemporarily (or entirely) made unavailable.

Intermediate system coordination engine 610 can monitor communicationchannels with intermediate systems to track the health and security ofthe channel. For example, a healthy connection can be inferred whenscheduled signals are consistently received. Further, intermediatesystem coordination engine 610 can track configurations of intermediatesystems (e.g., via communications generated at the intermediate systemsvia a software agent that identifies such configurations) so as toinfluence code generation, communication format, and/or provisions oraccess rights.

Thus, either via a presentation facilitated by publisher engine 608(e.g., via a web site or app page) or via communication with anintermediate system, a request for assignment of an access right can bereceived. A request management engine 612 can process the request.

Processing the request can include determining whether all otherrequired information has been received, such as user-identifyinginformation (e.g., name), access-right identifying information (e.g.,identifying a resource and/or access-right characteristic) user contactinformation (e.g., address, phone number, and/or email address), and/oruser device information (e.g., type of device, device identifier, and/orIP address).

When all required information has not been received, request managementengine 612 can facilitate collection of the information (e.g., via awebpage, app page or communication to an intermediate system). Requestmanagement engine 612 can also or alternatively collect paymentinformation, determine that payment information has been received,obtain authorization of payment, determine that payment has beenauthorized (e.g., via an intermediate system), collect payment, and/ordetermine that payment has been collected. For example, publisher engine608 may receive a credit card number and expiration date via a webpage,and request management engine 612 can request authorization for anamount of the requested access rights. In some instances, paymentassessments are performed subsequent to at least temporary assignmentsof access rights. In some instances, request management engine 612retrieves data from a user account. For example, publisher engine 608may indicate that a request for an access right has been received whilea user was logged into a particular account. Request management engine612 may then retrieve, for example, contact information, deviceinformation, and/or preferences and/or payment information associatedwith the account from account data store 324.

In some instances, request management engine 612 prioritizes requests,such as requests for overlapping, similar or same access rights (e.g.,requests for access rights associated with a same section) receivedwithin a defined time period. The prioritization can be based on, forexample, times at which requests were received (e.g., prioritizingearlier requests), a request parameter (e.g., prioritizing requests fora higher or lower number of access rights above others), whetherrequests were received via an intermediate system (e.g., prioritizingsuch requests lower than others), intermediate systems associated withrequests (e.g., based on rankings of the systems), whether requests wereassociated with users having established accounts, and/or whetherrequests were associated with inputs indicative of a bot initiating therequest (e.g., shorter inter-click intervals, failed CAPTCHA tests,purchase history departing from a human profile).

Upon determining that required information has been received andrequest-processing conditions have been met, request management engine612 can forward appropriate request information to a resource schedulingengine 614. For a request, resource scheduling engine 614 can queryresource status data store 528 to identify access rights matchingparameters of the request.

In some instances, the request has an access-right specificity matchinga specificity at which access rights are assigned. In some instances,the request is less specific, and resource scheduling engine 614 canthen facilitate an identification of particular rights to assign. Forexample, request management engine 612 can facilitate a communicationexchange by which access right characteristics matching the request areidentified, and a user is allowed to select particular rights. Asanother example, request management engine 612 can itself select fromamongst matching access rights based on a defined criterion (e.g., bestsummed or averaged access-right ranking, pseudo-random selection, or aselection technique identified based on user input).

Upon identifying appropriately specific access rights, resourcescheduling engine 614 can update resource status data store 528 so as toplace the access right(s) on hold (e.g., while obtaining paymentauthorization and/or user confirmation) and/or to change a status of theaccess right(s) to indicate that they have been assigned (e.g.,immediately, upon receiving payment authorization or upon receiving userconfirmation). Such assignment indication may associate informationabout the user (e.g., user name, device information, phone number and/oremail address) and/or assignment process (e.g., identifier of anyintermediate system and/or assignment date and time) with an identifierof the access right(s).

For individual assigned access rights, an encoding engine 616 cangenerate an access-enabling code. The access-enabling code can include,for example, an alphanumeric string, a text string, a number, a graphic,a barcode (e.g., a 1-dimensional or 2-dimensional barcode), a staticcode, a dynamic code (e.g., with a feature depending on a current time,current location or communication) and/or a technique for generating thecode (e.g., whereby part of the code may be static and part of the codemay be determined using the technique). The code may be unique acrossall access rights, all access rights for a given resource, all accessrights associated with a given location, all access rights associatedwith a given time period, all resources and/or all users. In someinstances, at least part of the code is determined based on or isthereafter associated with an identifier of a user, user deviceinformation, a resource specification and/or an access rightcharacteristic.

In various embodiments, the code may be generated prior to allocatingaccess rights (e.g., such that each of some or all allocated accessrights are associated with an access-enabling code), prior to or whileassigning one or more access right(s) responsive to a request (e.g.,such that each of some or all assigned access rights are associated withan access-enabling code), at a prescribed time, and/or when the deviceis at a defined location and/or in response to user input. The code maybe stored at or availed to a user device. In various instances, at theuser device, an access-enabling code may be provided in a manner suchthat it is visibly available for user inspection or concealed from auser. For example, a ticket document with a barcode may be transmittedto a user device, or an app on the user device can transmit a requestwith a device identifier for a dynamic code.

Encoding engine 616 can store the access-enabling codes inaccess-enabling code data store 530. Encoding engine 616 can also oralternatively store an indication in account data store 524 that theaccess right(s) have been assigned to the user. It will again beappreciated that data stores 524, 526, 528, and 530 can be relationaland/or linked, such that, for example, an identification of anassignment can be used to identify one or more access rights, associatedaccess-enabling code(s) and/or resource specifications.

Resource scheduling engine 614 can facilitate one or more transmissionsof data pertaining to one or more assigned access rights to a device ofa user associated with the assignment. The data can include anindication that access rights have been assigned and/or details as towhich rights have been assigned. The data can also or alternativelyinclude access-enabling codes associated with assigned access rights.

While FIG. 6 depicts components of resource access coordinator module616 that may be present on an access management system 285, it will beappreciated that similar or complementary engines may be present onother systems. For example, a communication engine on a user device canbe configured to display presentations identifying access rightavailability, and a request management engine on a user device can beconfigured to translate inputs into access-right requests to send to anintermediate system or access management system.

Returning to FIG. 5, code verification module 518 (e.g., at a userdevice or client device) can analyze data to determine whether anaccess-enabling code is generally valid and/or valid for a particularcircumstance. The access-enabling code can include one that is receivedat or detected by mobile device 210. The analysis can include, forexample, determining whether all or part of the access-enabling codematches one stored in access-enabling code data store 530 or partthereof, whether the access-enabling code has previously been applied,whether all or part of the access-enabling code is consistent withitself or other information (e.g., one or more particular resourcespecifications, a current time and/or a detected location) as determinedbased on a consistency analysis and/or whether all or part of theaccess-enabling code has an acceptable format.

For example, access-enabling code data store 530 can be organized in amanner such that access-enabling codes for a particular resource, date,resource group, client, etc. can be queried to determine whether anysuch access-enabling codes correspond to (e.g., match) one beingevaluated, which may indicate that the code is verified. Additionalinformation associated with the code may also or alternatively beevaluated. For example, the additional information can indicate whetherthe code is currently valid or expired (e.g., due to a previous use ofthe code).

As another example, a portion of an access-enabling code can include anidentifier of a user device or user account, and code verificationmodule 518 can determine whether the code-identified device or accountmatches that detected as part of the evaluation. To illustrate, device200 can be a client device that electronically receives a communicationwith an access-enabling code from a user device. The communication canfurther include a device identifier that identifies, for example, thatthe user device is a particular type of smartphone. Code verificationmodule 518 can then determine whether device-identifying information inthe code is consistent with the identified type of smartphone.

As yet another example, code verification module 518 can identify a codeformat rule that specifies a format that valid codes are to have. Toillustrate, the code format rule may identify a number of elements thatare to be included in the code or a pattern that is to be present in thecode. Code verification module 518 can then determine that a code is notvalid if it does not conform to the format.

Verification of an access-enabling code can indicate that access to aresource is to be granted. Conversely, determining that a code is notverified can indicate that access to a resource is to be limited orprevented. In some instances, a presentation is generated (e.g., andpresented) that indicates whether access is to be granted and/or aresult of a verification analysis. In some instances, access grantingand/or limiting is automatically affected. For example, upon a codeverification, a user device and/or user may be automatically permittedto access a particular resource. Accessing a resource may include, forexample, using a computational resource, possessing an item, receiving aservice, entering a geographical area, and/or attending an event (e.g.,generally or at a particular location).

Verification of an access-enabling code can further trigger amodification to access-enabling code data store 530. For example, a codethat has been verified can be removed from the data store or associatedwith a new status. This modification may limit attempts to use a samecode multiple times for resource access.

A combination of modules 514, 516, 518 comprise a secure addressableendpoint agent 520 that acts as an adapter and enables cross-deviceinterfacing in a secure and reliable fashion so as to facilitateallocation of access-enabling codes and coordinate resource access.Secure addressable endpoint agent 520 can further generate a healthsignal that is transmitted to another device for monitoring of a statusof a communication channel. The health signal is optionally a shortmessage of a few bytes or many bytes in length that may be transmittedon a frequent basis (e.g., every few milliseconds or seconds). Acommunications manager 506 on the receiving device can then monitors thehealth signal provided by the agent to ensure that the communicationlink between the host server and device 500 is still operational.

In some instances, device 500 can include (or can be in communicationwith) a physical access control 532. Physical access control 532 caninclude a gating component that can be configured to provide a physicalbarrier towards accessing a resource. For example, physical accesscontrol 532 can include a turnstile or a packaging lock.

Physical access control 532 can be configured such that it can switchbetween two modes, which differ in terms of a degree to which useraccess to a resource is permitted. For example, a turnstile may have alocked mode that prevents movement of an arm of the turnstile and anunlocked mode that allows the arm to be rotated. In some instances, adefault mode is the mode that is more limiting in terms of access.

Physical access control 532 can switch its mode in response to receivingparticular results from code verification module 518. For example, uponreceiving an indication that a code has been verified, physical accesscontrol 532 can switch from a locked mode to an unlocked mode. It mayremain in the changed state for a defined period of time or until anaction or event is detected (e.g., rotation of an arm).

Device 500 can also include one or more environmental sensors 534.Measurements from the sensor can processed by one or more applicationmodules. Environmental sensor(s) 534 can include a global positioningsystem (GPS) receiver 535 that can receive signals from one or more GPSsatellites. A GPS chipset can use the signals to estimate a location ofdevice 500 (e.g., a longitude and latitude of device 500). The estimatedlocation can be used to identify a particular resource (e.g., one beingoffered at or near the location at a current or near-term time). Theidentification of the particular resource can be used, for example, toidentify a corresponding (e.g., user-associated) access-enabling code orto evaluate an access-enabling code (e.g., to determine whether itcorresponds to a resource associated with the location).

The estimated location can further or alternatively be used to determinewhen to perform a particular function. For example, at a user device,detecting that the device is in or has entered a particular geographicalregion (e.g., is within a threshold distance from a geofence perimeteror entrance gate) can cause the device to retrieve or request anaccess-enabling code, conduct a verification analysis of the code and/ortransmit the code to a client device.

It will be appreciated that environmental sensor(s) 534 can include oneor more additional or alternative sensors aside from GPS receiver 535.For example, a location of device 500 can be estimated based on signalsreceived by another receive from different sources (e.g., base stations,client point devices or Wi Fi access points). As another example, anaccelerometer and/or gyroscope can be provided. Data from these sensorscan be used to infer when a user is attempting to present anaccess-enabling code for evaluation.

It will also be appreciated that the components and/or engines depictedin figures herein are illustrative, and a device need not include eachdepicted component and/or engine and/or can include one or moreadditional components and/or engines. For example, a device can alsoinclude a user interface, which may include a touch sensor, keyboard,display, camera and/or speakers. As another example, a device caninclude a power component, which can distribute power to components ofthe device. The power component can include a battery and/or aconnection component for connecting to a power source. As yet anotherexample, a module in the application layer can include an operatingsystem. As still another example, an application-layer control processormodule can provide message processing for messages received from anotherdevice. The message processing can include classifying the message androuting it to the appropriate module. To illustrate, the message can beclassified as a request for resource access or for an access-enablingcode, an update message or an indication that a code has been redeemedor verified. The message processing module can further convert a messageor command into a format that can interoperate with a target module.

It will further be appreciated that the components, modules and/oragents could be implemented in one or more instances of software. Thefunctionalities described herein need not be implemented in separatemodules, for example, one or more functions can be implemented in onesoftware instance and/or one software/hardware combination. Othercombinations are similarly be contemplated.

Further yet, it will be appreciated that a storage medium (e.g., usingmagnetic storage media, flash memory, other semiconductor memory (e.g.,DRAM, SRAM), or any other non transitory storage medium, or acombination of media, and can include volatile and/or non volatilemedia) can be used to store program code for each of one or more of thecomponents, modules and/or engines depicted in FIGS. 5 and 6 and/or tostore any or all data stores depicted in FIG. 5 or described withreference to FIGS. 5 and/or 6. Any device or system disclosed herein caninclude a processing subsystem for executing the code. The processingsystem can be implemented as one or more integrated circuits, e.g., oneor more single-core or multi-core microprocessors or microcontrollers,examples of which are known in the art.

FIG. 7 illustrates a flowchart of an embodiment of a process 700 forassigning access rights for resources. Process 700 can be performed byan access management system, such as access management system 285.Process 700 begins at block 705 where resource specification engine 302identifies one or more specifications for a resource. The specificationscan include, for example, a time at which the resource is to beavailable, a location of the resource, a capacity of the resourcesand/or one or more entities (e.g., performing entities) associated withthe resource.

At block 710, resource-access allocation engine 604 allocates a set ofaccess rights for the resource. In some instances, each of at least someof the access rights corresponds to a different access parameter, suchas a different location (e.g., seat) assignment. Upon allocation, eachof some or all of the access rights may have a status as available. Asubset of the set of access rights can be immediately (or at a definedtime) assigned or reserved according to a base assignment or reservationrule (e.g., assigning particular access rights to particular entities,who may be involved in or related to provision of the resource and/orwho have requested or been assigned a set of related access rights.

At block 715, communication engine 606 transmits the resourcespecifications and data about the access rights. The transmission canoccur in one or more transmissions. The transmission can be to, forexample, one or more user devices and/or intermediate systems. In someinstances, a notification including the specifications and access-rightdata is transmitted, and in some instances, a notification can begenerated at a receiving device based on the specifications andaccess-right data. The notification can include, for example, a websitethat identifies a resource (via, at least in part, its specifications)and indicates that access rights for the resource are available forassignment. The notification can include an option to request assignmentof one or more access rights.

At block 720, request management engine 612 receives a request for oneor more access rights to be assigned to a user. The request can, forexample, identify particular access rights and/or access parameters. Therequest can include or be accompanied by other information, such asidentifying information. In some instances, the access management systemcan use at least some of such information to determine whether a fee forthe access rights has been authorized. In some instances, the request isreceived via an intermediate system that has already handled suchauthorization.

At block 725, resource scheduling engine 614 assigns the requested oneor more access rights to the user. The assignment can be conditioned onreceipt of all required information, confirmation that the accessright(s) have remained available for assignment, determining using datacorresponding to the request that a bot-detection condition is notsatisfied, fee provision and/or other defined conditions. Assignment ofthe access right(s) can include associating an identifier of each of theone or more rights with an identifier of a user and/or assignment and/orchanging a status of the access right(s) to assigned. Assignment of theaccess right(s) can result in impeding or preventing other users fromrequesting the access right(s), being assigned the access right(s)and/or being notified that the access right(s) are available forassignment. Assignment of the access right(s) can, in some instances,trigger transmission of one or more communications to, for example, oneor more intermediate systems identifying the access right(s) andindicating that they have been assigned and/or with an instruction tocease offering the access rights.

At block 730, encoding engine 616 generates an access-enabling code foreach of the one or more access rights. The code can be generated, forexample, as part of the assignment, as part of the allocation orsubsequent to the assignment (e.g., upon detecting that a user isrequesting access to the resource). Generating an access-enabling codecan include applying a code-generation technique, such on one thatgenerates a code based on a characteristic of a user, user device,current time, access right, resource, intermediate system or othervariable. The access-enabling code can include a static code that willnot change after it has been initially generated or a dynamic code thatchanges in time (e.g., such that block 730 can be repeated at varioustime points).

At block 735, communication engine 606 transmits a confirmation of theassignment and the access-enabling code(s) in one or more transmissions.The transmission(s) may be sent to one or more devices, such as a userdevice having initiated the request from block 720, a remote server oran intermediate system having relayed the request from block 720.

Referring to FIG. 8A, an embodiment of a site system 280 is shown inrelation to mobile devices 824-n, Network Attached Storage (NAS) 850,site network 816 and the Internet 828. In some embodiments, forattendees of a live event or concert, site network 816 and site system280 provide content, services and/or interactive engagement using mobiledevices 824. Connections to site system 280 and site network 816 can beestablished by mobile devices 824 connecting to access points 820.Mobile devices 824 can be a type of end user device 110 that isportable, e.g., smartphones, mobile phones, tablets, and/or othersimilar devices. Mobile devices 824 are described in further detailherein with the description of FIG. 9.

Site network 816 can have access to content (information aboutattendees, videos, pictures, music, trivia information, etc.) held byNAS 850. Additionally, as described herein, content can be gathered fromattendees both before and during the event. By connecting to sitenetwork 816, mobile device 824 can send content for use by site system280 or display content received from NAS 850.

Referring to FIG. 8B, another embodiment of a site system 280 is shownin relation to mobile devices 824-n, Network Attached Storage (NAS) 850,site network 816 and the Internet 828, in an embodiment. FIG. 8Badditionally includes phone switch 840. In some embodiments, phoneswitch 840 can be a private cellular base station configured to spoofthe operation of conventionally operated base stations. Using phoneswitch 840 at an event site allows site system 280 to provide additionaltypes of interactions with mobile devices 824. For example, without anysetup or configuration to accept communications from site controller812, phone switch 840 can cause connected mobile devices 824 to ringand, when answered, have an audio or video call be established. Whenused with other embodiments described herein, phone switch 840 canprovide additional interactions. For example, some embodiments describedherein use different capabilities of mobile devices 824 to cause masssounds and/or establish communications with two or more people. Bycausing phones to ring and by establishing cellular calls, phone switchcan provide additional capabilities to these approaches.

FIG. 9 illustrates example components of a mobile device 824. Mobiledevice 824 can connect to access point 820 using one or more antennas959, and wireless modules 950 (e.g., WiFi 952, Bluetooth 954, NFC 956and/or cellular 958). Once connected to site network 816 and sitecontroller 812, handheld controller 910 can use input and outputhardware components and/or sensors to enable different embodimentsdescribed herein.

Input sensors used by mobile device 824 described herein include,accelerometer 932, gyroscope(s) 934, light sensor 936, magnetometer(e.g., compass) 938, and barometer 939. Location engine 937 can usegeolocation hardware components (e.g., wireless signal receivers,iBeacon, NFC, Global Positioning System (GPS), and/or other similarcomponents). Peer monitor 930 uses a data structure to store and updatesthe location of nearby mobile devices 824.

In some embodiments, peer monitor 930 receives information from sitecontroller 812 that corresponds to the locations of other nearby mobiledevices. In embodiments, peer monitor can also use sensors to locatenearby mobile devices 824 (e.g., Bluetooth 954, NFC 956, and or othersimilar sensing hardware). Other input components used by someembodiments include microphone 972, camera 976, and front-facing camera980, respectively controlled and/or providing capture by audio capturemodule 974 and camera capture module 978. One having skill in therelevant art(s), given the description herein, will appreciate thatother input and or sensor components can be used by embodiments ofmobile device 824.

In some embodiments, components are included that assist with theprocessing and utilization of sensor data. Motion coprocessor 915, 3Dengine 916, and physics engine 917 can all process sensor data, and alsoperform tasks graphics rendering related to graphics processor 968.

Output components used by some embodiments include speaker 962, display966, LED 940, and flash 942, respectively controlled and/or relayedoutput information by, audio engine 964, graphics engine 968 and screencontroller 970, LED controller 964, and flash controller 946. Otheroutput components used by mobile devices 824 include NFC 956 andBluetooth 954, which, beyond wireless communication capabilities canalso be used to send instructions to certain devices.

Some embodiments described herein use information collected by mobiledevice 824 using connections to site controller 812 and/or site network816. This information collected (e.g., pictures, video, recorded audio,movement data collected by gyroscope 934 and/or accelerometer 932,and/or other collectable data) can be stored by mobile device 824 inuser storage 924. In addition, in some embodiments described herein,site controller 812 can control mobile device 824 by sending controlsignals to the device, and storing information in application storage922 (e.g., content to be displayed, hardware configuration instructions,trigger commands for hardware components, and/or other similarinformation). In some embodiments, handheld controller 910 uses storagecontroller 920 to store and retrieve information from applicationstorage 922 and user storage 924. Storage controller 920 can alsoretrieve executable code from code storage 926 for execution bydifferent processor components.

One having skill in the relevant art(s), given the description herein,will appreciate that other combinations of similar components can beused to provide the features of the components described above, e.g.,components described above could be replaced by components that havebeen combined into integrated components and/or components that havebeen divided into multiple components, e.g., a variation of camera 976can include a camera capture module 978 as an integrated unit.Additional descriptions of example components used by mobile device 824are provided below.

FIG. 10 illustrates sample components of an embodiment of site system280, including connections to NAS 850 and access management system 285.Embodiments of site controller 812 use network manager 1020 to connectvia access points 820 (using e.g., WiFi 1052, Bluetooth 1053, NFC 1056,Ethernet 1058, and/or other network connections) to other networkcomponents, such as site network 816 and mobile devices 824. In someembodiments, site system 280 uses site controller 812 to control aspectsof an event venue. A broad variety of venue features can be controlledby different embodiments, including: permanent lights (e.g., withlighting controller 1022), stage lights (e.g., with presentmentcontroller 1024), stage display screens (e.g., with stage display(s)controller 1012), permanent display screens (e.g., with permanentdisplay(s) controller 1014), and the venue sound system (e.g., with thesound system controller 1016). In an embodiment discussed further withthe discussion of FIG. 27 below, different presentation devices (e.g.,an LED sculpture) can present information from site controller 812.

A more detailed view of NAS 850 is shown, including NAS controller 1030coupled to user video storage 1032, captured video storage 1034,preference storage 1036, and 3D model 1038. Captured video storage 1034can receive, store and provide user videos received from mobile devices824. In some embodiments, site controller 812 triggers the automaticcapture of images, audio and video from mobile devices 824, suchtriggering being synchronized to activities in an event. Images capturedby this and similar embodiments can be stored on both the capturingmobile device 824 and user video storage 1032. In an embodiment, sitecontroller 812 can coordinate the transfer of information from mobiledevices to NAS 850 (e.g., captured media) with activities taking placeduring the event. When interacting with mobile devices 824, someembodiments of site controller 812 can provide end user interfaces 1026to enable different types of interaction. For example, as a part ofengagement activities, site controller may offer quizzes and othercontent to the devices. Additionally, with respect to locationdeterminations discussed herein, site controller can supplementdetermined estimates with voluntarily provided information using enduser interfaces 1026, stored in a storage that is not shown.

In some embodiments, to guide the performance of different activities,site controller 812 and/or other components may use executable code 1038tangibly stored in code storage 1039. In some embodiments, siteinformation storage 1037 can provide information about the site, e.g.,events, seat maps, attendee information, geographic location ofdestinations (e.g., concessions, bathrooms, exits, etc.), as well as 3Dmodels of site features and structure.

With reference to FIG. 11, an embodiment of a wireless topology 1100 isshown. In this example, access points 820 are divided throughout a venuewith WiFi hotspots 1104. In other embodiments, Bluetooth™, cellular,WiMax™, Zigbee™, or other base stations or hotspots could be usedalternatively or in combination. Mobile devices 1124 can move throughoutthe venue and connect to various hotspots. Location estimations used bysome embodiments described herein can be performed by triangulatingbetween multiple hotspots with known locations. Based upon whichhotspots are visible, and the measured strength of respective signals,mobile devices 1124 can estimate location. Location can also beprovided, or the previously described approach can be enhanced by, othergeolocation approaches such as ranging, GPS and/or other mechanisms.

FIG. 12 illustrates a flowchart of an embodiment of a process 1200 forinitializing contact with a mobile device 824 and estimating thelocation of the mobile device for use with other processes. In block1210, event tickets are allocated. Other information that can becaptured and stored at the time of ticket purchase include: one or morenames of people who will be using purchased tickets, the home address ofthe purchaser, and/or other similar information.

In block 1220, the determined unique identifier is stored as associatedwith one of the allocated tickets. In some embodiments, a mobile device824 can be used at the time of purchase of a ticket, and, because ofthis, a unique identifier for the mobile device 824 can be detected andstored as associated with a ticket. At block 1220, this uniqueIdentifier is associated with an identifier from the redeemed ticket isstored.

At block 1225, at the time of an event, a ticketholder redeems theticket to enter the event venue site. As described above with thedescriptions of FIGS. 2 and 3, access management system 285 manages theissuance and redemption of tickets for some embodiments.

Once at the site, different wireless communication techniques can beused to detect any mobile devices carried to the event and connect themto the site network. For example, at the time the ticketholder redeemstheir ticket, their mobile device 824 can have WiFi enabled. Usingaccess point 820 with this WiFi protocol, a network connection can beestablished between the mobile device 824 of a ticketholder and sitenetwork resources such as site controller 812 and site network 816.Other connections can also be used to connect to access point 820,including Bluetooth 1053 and NFC 1056.

In some embodiments, based on the connection to access point 820, auniform identifier of mobile device 824 can be determined, and thisidentifier can be used to retrieve information about the ticketholder.Thus, at block 1230, a unique identifier of mobile device is associatedwith the ticket used by the ticketholder to gain entry to the venue.Alternatively, if mobile device 824 was used to present an e-Ticket forredemption, this uniform identifier could be determined at the time ofredemption. An embodiment can automatically establish a connectionbetween a mobile device 824 and access point 820 at the time anelectronic ticket is redeemed using the mobile device.

In some embodiments, the connection established between mobile device824 and a ticket used to enter the venue can be used to improve thegeolocation capabilities of mobile device 824. Based on informationassociated with the redeemed ticket, stored seat information isidentified and retrieved. In block 1240, the location of the mobiledevice is estimated based on the on seat information, e.g., locationcoordinates for each seat at a venue can be stored and available in theNAS 850 for use by embodiments. Having this initial location estimatecan improve the geolocation process by improving the speed of initialestimates (e.g., the database retrieval of the location of a seat isgenerally faster than establishing GPS satellite connections).

Continuing with the geolocation (and/or indoor location) of mobiledevice 824, at block 1245, secondary location information is requestedfrom the mobile device. In some embodiments, this secondary locationinformation includes GPS data for a slower completing, but more accurateestimate. At block 1245, the secondary location information is receivedfrom mobile device 824.

In block 1255, the received secondary location information is comparedwith seating information for the venue. In some embodiments, thisseating information can be plotted in a model (e.g., a 3D model storedin site information storage 1037 in NAS 850). Based on this comparison,at block 1260 the mobile device is present at the ticketed seatinglocation (e.g., comparing received GPS coordinates with the 3D Model).At block 1261, when the mobile device is estimate to be in the seatingposition, this estimated position is stored for further use with thisdevice.

At block 1265, when it is estimated that mobile device 824 is not at alocation associated with the ticketed seat on record for the device,site controller 812 requests accelerometer, gyroscope and/or compassdata. In block 1270, the movement sensor information is received, and inblock 1265, site controller 812 can use this information provided bymobile device 824 to classify the movement type estimated as beingperformed with mobile device 824. Because embodiments described hereinoperate in venues for concerts, some embodiments do not automaticallyequate movement with locomotion. At block 1280, if, after analysis ofthe collected movement data indicates that mobile device is not beingtransported (e.g., the person holding is likely dancing, not walking atthe venue). At block 1285, once it is likely that mobile device 824 isnot being transported, a new location estimate is performed based oncollected information, and this location estimate is used in additionalprocesses described herein.

It should be appreciated that many of the additional locationdetermining blocks described with FIG. 12 are optional, and someembodiments can determine location by a single approach, e.g., iBeacon,wireless geolocation, or GPS.

FIG. 13 illustrates a flowchart of an embodiment of a process 1300 forlinking mobile devices at an event using groups based on seating andactivity. At block 1200, the initialization and location estimateprocess is performed. As is described above, in some embodiments, mobiledevices 824 can be associated with people. Social networks rely uponlinking people based on shared characteristics. In block 1320,information associated with mobile devices (e.g., information about theassociated owners of the devices) can be retrieved from data storesmaintained by server systems associated with generating and maintainingtickets (e.g., access management system 285 and site controller 812).

In block 1325, the retrieved data from the ticketing systems can be usedto retrieve additional information from independent sources. Forexample, when a mobile device 824 is associated with a ticket, and theticket is associated with a particular person, the name of that personcan be used to search online social networks. In some embodiments,ticket purchasers provide credentials to their social networking siteswith the understanding that information from these sites will be used toimprove a concert experience. These credentials can be stored for use byembodiments, either in storage on mobile device 824 (e.g., applicationstorage 622) or on servers available to site system 280 (e.g., in NAS850 for use by site controller 812).

Searching operations can be performed by embodiments using a variety ofdifferent components. Based on access to Internet 528, NAS 850 andaccess management system 285 (e.g., for information about ticketpurchasers), site controller 812 performs the searching, analysis forsome embodiments of this section.

At block 1330, the collected results of queries performed for multiplepeople are analyzed and, at block 1335, mobile devices 824 are groupedbased on the retrieved data. For example, a search of popular socialnetwork sites reveals that several event attendees are avidskateboarders, college professors and/or like falafel. Adding an extradimension, to promote the formation of closer groups, some embodimentsfactor in the seating arrangement in the venue, with distance apart,level of seating, and/or other seating oriented factors influencing theorganization of event attendees into groups. At block 1335, these groupsare formed automatically by embodiments.

Once groups are formed, the mobile devices for each member of the groupis notified of the group, and different features can be explained. Insome embodiments, site controller 812 uses contact information stored inNAS 850 to generate messages for group members. With respect tomessaging, embodiments can combine standard approaches (e.g., text,audio, video) on the phone, with features that are only available toevent sites. For example, text messages notifications can be sent usingthe venue video displays, LED sculptures and other similar approaches.Video messages announcing groupings can include members of a performingband and/or celebrity attendees. Other approaches would be apparent toone having skill in the relevant art(s), given the disclosure herein. Insome embodiments, groups would be able to easily communicate using avariety of different approaches.

In an embodiment, site controller 812 can use phone switch 140 to causean attendee's phone to ring or receive video call using a video callingservice. In some embodiments, a call could be connected with anotherperson in the venue or in the attendee's social graph who is inside oroutside the venue. The identity of the call connection and originationlocation is unknown in one embodiment until the receiver says hello,hangs up or begins a conversation. In different embodiments, All overthe venue attendees could be connected at random.

Continuing this example, after a minute or some other predetermined timeof conversation, phone switch 840 can call a third person and patchesthem into the conversation. Then maybe a fourth, a fifth, or perhaps oneof the original participants is bumped. The audio and/or video callscould be recorded and stored on NAS 850 for later use at the venue oranother stop on the tour. The connected callers could be nearby, perhapsin adjacent seats or rows such that the call is an “ice-breaker” tofacilitate meeting in person. The callers could be given the option tomeet one or more persons on the call. Some embodiments could evenfacilitate a seat swap so that newly introduced attendees can changeseats. Members of the band, performers or venue staff could even get ona call optionally. Phones could ring or calls could be terminated insynchronization with the performance. A message could interrupt the callto transition to the performance or otherwise notify the attendees ofsomething.

In some embodiments, a phone video and/or audio could be patcheddirectly to a stage display. The video camera on the mobile device 824could be used to create a virtual “fan cam” for display on the stagedisplay, thus providing display video without needing to sendprofessional camerapersons into the crowd.

In an example, anybody attending a show at a venue can receive a phonecall such that one doesn't need an app or a smartphone to be included.Calls can be received in the venue, for example, the concessions line,in the bathroom, in a venue bar, outside in the parking lot, in line toenter. Embodiments can ring any phone number that was associated with anattendee. Some embodiments could confirm that the registered phonenumber was at the show before connecting.

To promote communication between and among individuals within the formedgroups, at block 1340, two members at a time are selected in a singlegroup for pairing analysis. In an embodiment, this analysis is relatedto the event to which the audience is attending. One person may havebeen to recent concerts by this artist, and another person has notattended any concerts.

The purchase of souvenirs at shows can also be measured and comparedbased on credit card receipts. Interest in an artist can also bemeasured by searching for blog posts and other Internet information. Insome embodiments different types of connection analysis can be used,e.g., similarity in general interests, like of different foods or sportsteams, and/or other similar factors.

In block 1340, an affinity score is generated based on the types ofanalysis described above, and at block 1342, if the affinity scoreexceeds a threshold, communication links are established between the twoanalyzed people. In some embodiments, to promote connections betweensimilar audience members, automatic communications are generated fromone of the linked pair to another. These communication can be timed tosynchronize with activities taking place within the show, e.g.,different bands playing, different songs playing, and/or a guitar ordrum solo.

As described with block 1350, some embodiments can select the method andtiming of communications between pairs based on the information learned.For example, pairs judged to have a high affinity may prefer video callsover audio calls. In-person meetings can also be suggested at an earlytime. In block 1355, the communication is enabled between the pairs.

It should be appreciated that some embodiments can use an approach togrouping and connecting individuals that is less complicated than theprocesses described above. An embodiment can connect users in sitenetworks using basic information about the individuals and estimatedlocation.

An example method of method of linking mobile devices at an event usinggroups based on seating and activity includes the following:

-   -   receiving a first indication that a first ticket for an event        has been redeemed, the first ticket being associated with a        first seat at the event;    -   receiving a second indication that a second ticket for the event        has been redeemed, the second ticket being associated with a        second seat at the event, the first and seconds seats being        different seats;    -   retrieving first information associated with a first mobile        device, the first mobile device being associated with the first        ticket;    -   retrieving second information associated with a second mobile        device, the second mobile device being associated with the        second ticket;    -   analyzing the first and second information;    -   grouping the first and second mobile devices into a first group        based on the analyzing and respective locations of the first and        second seats;    -   establishing a first link between the first and second mobile        devices based on the analyzing; and    -   based on an activity at the event, enabling a first        communication between the first and second mobile devices based        on the first link.

Referring to FIG. 14 a diagram of a seating arrangement 1400 in a venueis shown.

Groupings shown with dotted lines (e.g., 1460-n) demonstrate thecapability of the generated groups to foster interactions amongdifferent attendees (e.g., persons 205-n). Some embodiments addadditional factors to the grouping approaches discussed with FIG. 13above, adding criteria that reflects a seating diversity, e.g., seatsfrom different sections, different types of seats (e.g., raised orfloor), whether a mobile device is associated with a seat at all (e.g.,standing room only admission), can all be evaluated by some embodimentsto advantageously group users.

For example, group 1460-3 integrates those from a raised section (e.g.,user 205-5) and a user in floor section 1440 (e.g., user 205-7). Inanother example, the flexibility of the system allows User 205-6standing on walkway 1435 to be in a group with seated user 205-1. Ingroup 1460-4, two people with common interests—one standing (205-6) inan isle (1435) and one sitting in a seat (205-1) can be linked into anaffinity group, and connected using approaches described with FIGS.13-15.

FIG. 15 shows a timeline 1500 that illustrates how some embodiments canuse event activities to influence the timing of contact between twoattendees: Users 205-10 (A) and 205-11

(B). In this example, an event begins at point 1520 where doors open tolet attendees in. Soon after, User A redeems his ticket 1594-1 forentry, with access management system validating the ticket code andissuing instructions to a computerized turnstile. While walking aroundthe venue, mobile device 824-10, which User A carries, automaticallyconnects to access points 820 using WiFi. Access points 820 reads theMedia Access Control (MAC) Address of mobile device 824-10 and relaysthis uniform identifier to site controller 812. Site controller 812queries access management system 285 to determine whether the MACaddress is associated with a ticket. Because User A purchased his ticketusing mobile device 824-10, this MAC address is stored as associatedwith the ticket. Because the ticket is associated with his identity,mobile device 824-10 is now associated with his identity.

In the meantime, User B uses her mobile device 824-11 to redeem anelectronic ticket 1594-2. Because she redeemed an electronic ticket withmobile device 824-11, this mobile device is connected to site network816 when the electronic ticket was redeemed 1596-2. While both Users Aand B are walking around, Users C-F also have their devices join sitenetwork 816. Based on similar characteristics, users A-F are groupedinto a group (1550) based on data retrieval and analysis performed bysite controller 812. Users A-F all get a notification message that theyhave been linked in a group of five attendees. A map shows them whereeach member of the group is, and some selected media from other socialnetworking accounts for several of the users.

As described with FIG. 13 above, based on the established group, anadditional analysis is can be performed regarding an affinity score formember of a group, e.g., Users A and B. After this analysis, while bothusers are sitting in their ticketed seats, an additional notificationarrives telling them that they have been specially linked 1555. User Auses the opportunity to send a text message to User B at point 1560,then both users settle into watching the show. At point 1565 however,both of their phones ring to connect them. As they answer, they bothhear the lead guitarist for the band they are watching thank them forattending, and tell them to get ready for his big guitar solo at point1545.

It should be appreciated that some embodiments are able to synchronizeautomatic mobile device functions with both scheduled and ad-hocactivities in an event. Scheduled activities rely upon a synchronizationof clocks between site system 280 servers and mobile devices 824. Ad-hocevents can be used to control the operation of system components (e.g.,messages sent by site controller 812) by detectable triggering events(e.g., by particular sounds, and/or lights). Thus, with regard to theexample above, the guitar solo could not be scheduled, but a particularcombination of sounds leading up to the solo in the song could bedetected.

Continuing with FIG. 16, this figure shows how affinity groups describedabove with FIG. 13 can be used to intuitively link people. In anexample, after users are automatically linked into one or more groupsbased on approaches similar to the ones described with FIG. 13 above,slider 1605 can be used to select a range of people over a spectrum ofinclusion 1670.

When a user wants to select a group to receive a group communication,the slider 1605 interface can improve the operation of this selection ofrecipients. In some embodiments, by linking the sharing of content tophysical space and the organization of recipients within the space(e.g., people in a section, people around you), the slider is rooted tothe physical world. With its predictable delivery of messages within aset space, it provides clear, tangible results. As indicated on theslider settings, the device allows for the selection of ever wider ornarrower groups of recipients. This interface feature can be combinedwith the grouping based on seats, activity and affinity described withthe description of FIG. 13.

FIG. 17 illustrates a flowchart of an embodiment of a process 1700 forusing each display of multiple mobile devices to display a single imageat an event. The depicted portion of the process begins in block 1710,where the user purchases a ticket using an app, web page or physicallocation. The ticket is loaded onto the app to become an electronicticket or at least record the exact location of the seat and row. Inblock 1720, the user can optionally record the experience or theirthoughts in buying a ticket with sound, video and/or a photo.

In one embodiment, the mobile devices 824 are transformed into a visualcanvas within the venue. To identify location, seat location could beused as gathered from the electronic ticket or manually entered, or theGPS signal on the mobile device 824, or ranging from wireless or soundsignals. For example, the microphone on the end user device could beused to record sound signals from the performance or test tones that theapp could use to triangulate location in the venue.

In various embodiments, many different visual effects can beaccomplished by embodiments within the venue, for example:

-   -   Run color through the screen of all mobile devices 824 in the        arena    -   Paint legible bold text at large scale across the arena—using        each mobile device 824 or cluster of mobile devices 824 as a        ‘pixel’ for dot matrix lettering    -   To control the flashlights or LEDs on each mobile device 824 to        create bold simultaneous flashes of light throughout the        audience    -   Turn on and off screens in synchronization or status lights/LEDs

In some embodiments, the location uncertainty of mobile devices 824 canbe quantified and used to select and/or modify the display of images.For example, for locations that are very precise down to the seat, finergrained effects could be used such as a pixel in an image or text.

Where estimated locations of mobile devices are less precise, colorchange, pulsating or modulating effects across the entire venue can beused. At any given moment, mobile devices 824 could be grouped andassociated with different effects according to their location accuracyto create a chorus when done in sync. Effects that work best with acertain pixel density could avoid areas of the venue where that doesn'texist or could be postponed, canceled or substituted with anothereffect.

An additional effect that can be generated by embodiment is a wave ofsound and color to compliment a wave of attendees standing insynchronization. The flash feature of the phone could be used to markthe creshendo of the wave as it circles the venue. The vibration featureof the phone could indicate to the attendee exactly when to stand insyncrhonization with the wave. Optionally, a photo could be taken whenthe flash is activated for upload and storage on the NAS 850 to enhancethe content available at the venue.

Returning to the discussion of process 1700, on the day of the event,the user checks into the event in block 1725. Checking in could bethrough an entrance or automatically as the electronic ticket is used.For example, when in contact with the site network 816, the ticket couldbe marked as used. Mobile device 824 can be connected, in block 1730 tosite network 816. In block 1735, predetermined pixel effects are loadedinto the mobile device 824. The pixel effects include things like colorchanges for the screen, flash activation sequences, sounds, video,and/or vibration sequences. The clock on the mobile device 824 issynchronized to the site network or an offset from an absolute referencein the venue is stored on the mobile device 824. In this embodiment,synchronizing allows the pre-loaded pixel effects to happen at apredetermined time.

A determination of location is shown in block 1740. This determinationcan be recurring and adjusted as the user might move throughout thevenue. Some embodiments calibrate the pixel effects in block 1750. Avideo camera or microphone in the venue could be used to receivefeedback from pixels when they activate a pixel effect. Should a pixelbe out of place, the location could be adjusted to correct for thelocation error. The brightness and sound amplitude could be adjustedalso. There could be calibration points distributed throughout the venueor other pixels could be used to gather that information. For example,in a small area one phone could have its speaker emit a tone that isrecorded through the microphone of several other phones to verify thelocation of the both the emitting phone and those receiving the sound.

At various times during the event the pixel effects are performed inblock 1750. Activation of a pixel effect is triggered by identifying theeffect and providing a synchronized time for performing the pixeleffect. In block 1780, content is captured on the mobile device 824 atthe option of the user or at the command of the site controller 812. Inblock 1765, live or stored content is downloaded from the site networkto the end user device for playback immediately or at a later time. Someembodiments could have the content played in synchronicity with the liveperformance, for example a slide show of photos gathered inside oroutside the event could be played to accompany the performance. Livecontent can also be captured with another pixel or by the venue forplayback with the mobile device 824 in block 1770.

The performer can optionally interact live with the pixels in block1775. For example sweeping their hand across the crowd could cause awave of color, sound and/or vibration to play on the mobile device 824.Content or information from the venue could be forwarded to each user'ssocial media accounts in block 356. After the event, the site network816 and the content stored on the NAS 850 can be accessed by attendees.For example, videos and photos gathered at the event are accessiblethrough the Internet 528. As the performers tour, site network 816 canbe moved in whole or part to the next venue.

FIG. 18 illustrates a flowchart of another embodiment of a process 1800for using each display of multiple mobile devices to display a singleimage at an event. At block 1200, the initialization and locationestimate process is performed. In block 1815, an image for display at anevent using multiple devices is selected. In block 1820, site controller812 accesses a three-dimensional model of the venue (e.g., 3d model1038) and transforms the placement of people, mobile devices and seatsinto a plan to select multiple mobile devices for display of theselected image. The selection of the multiple mobile devices is based onthe estimated location of each mobile device, and characteristics ofdisplay image.

In block 1825, ambient characteristics associated with display by theselected mobile devices are collected. In block 1830, the collectedcharacteristics, the estimated locations of the selected mobile devices,and the selected display image are analyzed. In block 1835, the selecteddisplay image is divided into smaller images based on the analysis. Inblock 1840 each smaller image is assigned to one of the selected mobiledevices. In block 1845, the smaller images can be transformed based oncollected ambient characteristics associated with the mobile device towhich the image is assigned. In block 1850, the transformed smallerimages are sent to the mobile devices to which they are assigned. Inblock 1855, an indication is sent to each of the selected mobile devicesto collectively show the display image.

Referring to FIG. 19, an example of the process, as described with FIG.18 above, for using each display of multiple mobile devices to display asingle image at an event is shown. FIG. 19 shows sections of seating ina venue. In this example, each white dot (e.g., 205-20) represents aseat having a user 205, and each black dot represents a user (e.g.,205-22), holding up a mobile device 824 that is engaging in thecollective display of an image or video. In this example, a video of arunning man is represented, with two stick figures showing how animationcould be achieved. It should be appreciated that the resolution of imageand video displays, with this approach is only limited by the density ofpeople and/or he size of the mobile device screens used (e.g.,smartphone, tablet and/or other devices described herein).

As discussed above with FIG. 18, some embodiments of this collectivedisplay approach can measure ambient characteristics of the displayenvironment and transform the display characteristics of differentsmaller display images to compensate. This measurement and adjustment isillustrated by area 1930 in FIG. 19, as representing an area where thedisplay can be improved by adjustments. The adjustments are illustratedby the grey dots (e.g., 205-23), these dots having an adjusted color todisplay better in area 1930. As noted with the discussion of FIG. 18above, cameras can be used to measure the quality of the collectiveimage display, and calibrations can be made. By sampling the displayfrom different distances and angles, calibration cameras 1950-1, 1950-2,and 1950-3 can be used to implement the adjustments described above.

FIG. 20 illustrates a flowchart of an embodiment for using activitiesand locations associated with tickets to display images received frommultiple mobile devices at an event. In addition to emitting audio andvisual information as a contribution to the event, information could becaptured by the audience mobile devices 824, and through the sitenetwork 816, and emitted beyond the venue—to extend the reach of theexperience. In various embodiments, content can be gather in any numberof ways using the app on the mobile device 824, for example:

-   -   Mobile devices 824 take a photograph simultaneously—the        photographs are stitched together to form a ‘mass photograph’ of        the event—a cubist fractured image from multiple angles—which is        transmitted to NAS 850 or shared through the Internet 128,        e-mail, and/or social networking outlets.    -   Multiple Mobile devices 824 can be triggered to take a        ‘selfie’—the photographs can then be stitched together to form a        ‘mass photograph’ of the audience—this becomes a cumulative mass        photograph—which is transmitted to the storage array 132 or        shared through the Internet 128, e-mail, and/or social        networking outlets.    -   Mobile devices 824 record its owner singing at the concert—the        recordings are layered together to form a mass choral track—this        becomes a cumulative mass chorus recording—being added to each        night of the tour—which is transmitted to the storage NAS 850 or        shared through the Internet 128, e-mail, and/or social        networking outlets.

At block 1200, the initialization and location estimate process isperformed. At block 2015, multiple activities associated with an eventare monitored. In an embodiment, site controller 812 can monitor theoccurrence of different events, such as different bands playing, or theoccurrence of particular sounds. In block 2020, an activity is selected,and in block 2025, a location is identified that is associated with theselected activity. I embodiments, site controller 812 can utilize 3Dmodel 1038 to provide location information for activities monitored byvideo.

In block 2030, ambient characteristics associated with the location arecollected, and the characteristics are analyzed along with the estimatedlocations of the mobile devices available for collection. In block 2040,mobile devices as selected based on this analysis. In block 2045, thecollected ambient characteristics are compared to a threshold, and, ifthe threshold is exceeded, in block 2050, modified capture settings arecreated based on the analysis. In block 2055, the modified capturesettings are sent to the selected mobile devices.

At block 2060 (also the block that would have executed if the thresholdhad not been exceeded), the trigger capture is triggered in the selectedmobile devices. At block 2065, the captured images are displayed, and atblock 2070, the captured images are displayed.

FIG. 21 shows an overhead view of a stadium 2100 having a stage 2120,and two stadium video displays (2130-1 and 2130-2). Two users, 205-12and 205-13 are shown at opposite ends of the stadium. The description ofFIG. 22 below describes the interaction of these elements with sitesystem 280, according to an embodiment.

FIG. 22 shows a timeline 2200 that illustrates how some embodiments canuse show activities to automatically trigger the capture of media bymultiple mobile devices. An event in stadium 2100 begins at 2220 wheredoors open to let attendees in. Soon after, Users 205-12 and 205-13redeem their tickets 2294-3, 2294-4 for entry, with access managementsystem 285 validating the ticket code and issuing instructions to acomputerized turnstile. Both users connect to access points 820 usingWiFi and take their seats as depicted in FIG. 21.

At the beginning of the main act (2235), in an embodiment, sitecontroller 812 begins to send control signals to mobile devices 824-10and 824-11, held respectively by Users 205-12 and 205-13. Sitecontroller has selected these two devices for the capture of images fordisplay during the concert. This selection can be made based on one ofmore factors that can affect the image capture (e.g., location in thevenue, age of the mobile device holder, type of mobile device to beused, and/or other similar characteristics). The control signals arecontrolling both camera settings (e.g., exposure, shutter speed, focus)and the triggering of the camera capture.

After the main act begins at 2235, both users stand up and have theirdevices pointing out, such that their camera lenses are facing thestage. In some embodiments, site controller 812 can send control signalsthat cause the mobile device to request this camera position. In someembodiments, at a given time, multiple mobile device cameras may beavailable and controlled by site controller 812, but only a portion areused to collect images. The automatic selection of cameras can beaffected by many factors, including: the type of mobile device andquality of camera, the angle at which the camera is held, the vantagepoint of the camera, and/or other similar features.

Continuing this example, at point 2250, where the singer stands stageright to address the crowd, both mobile devices held by users 205-1 and13 are held ready to capture. As FIG. 21 depicts however only User205-12 is in a position to take a picture. When triggering capture, sitecontroller 812 can use image parsing technology to determine thecomposition of potential images. In this example, site controller 205-12only triggers the capture of an image for User 205-12.

In this example, stadium video displays (2130-1 and 2130-2) arepositioned such that User 205-12 can only see stadium video display2130-1 but not display 1830-2. As discussed herein, site controller 812maintains an estimate of different mobile device locations (the mobiledevice of user 205-12 was selected based, in part, on its position). Insome embodiments, site controller 812 can maintain a 3D model of stadium2100 (e.g., in site information storage 1037), and can estimate vantagepoints for different locations within the stadium. Based on thesecapabilities, for each picture captured to be displayed, site controller812 can determine which displays can be seen by which users. In someembodiments, to improve the mass collection and display of media,displays are selected based on promoting the most happiness from picturetakers. Thus, in this example, video display 2130-1 is selected todisplay the picture that was captured by user 205-12's mobile device.

With reference to FIG. 23 a flowchart of an embodiment of a process 2300that uses each speaker of multiple mobile devices to output sounds at anevent. FIG. 24 is an illustration of a stadium where an embodiment ofprocess 2300 is implemented during a musical performance from stage2440. Mobile device 824-1, 824-2, and 824-3 are used by embodiments. Tocalibrate the output of the multiple mobile devices 824, microphones2410A and 2410B are used.

At block 1200, the initialization and location estimate process isperformed. For example, a connection by mobile device 824-1 is receivedby access points 820, and the location of mobile device is estimated. Insome embodiments, site controller 812 uses the location estimates forthe available mobile devices to map them in a 3D model stored in siteinformation storage 1037. At block 2315, multiple related soundsassociated with the event are selected. In this example, the artistperforming at the concert select some high and low sounds that willcomplement the instruments heard through the concert sound system. Atblock 2320, multiple mobile devices 824 are selected from the availablemobile devices based on estimated mobile device location andcharacteristics of the selected related sounds. As for all embodimentsdiscussed herein that use capabilities of mobile devices, for thisanalysis, in some embodiments the model of the mobile device can betaken into account when selecting mobile devices for differentactivities.

At block 2325, ambient characteristics associated with the multiplemobile devices 824 are collected. At block 2330, the collected ambientcharacteristics, the estimated locations of the selected mobile devices,and the related sound are analyzed. In block 2335, the collected ambientcharacteristics are compared to a capture quality threshold, and, if thethreshold is exceeded, in block 2340, modified capture settings arecreated based on the analysis. In block 2345, the modified capturesettings are sent to the selected mobile devices. At block 2350 (alsothe block that would have executed if the threshold had not beenexceeded), each of the related sounds are sent to one or more of theselected plurality of mobile devices.

In some embodiments, microphones 2410-1 and 2410-2 can be used tomeasure the quality of the sounds generated by this process. Havingmultiple calibration microphones not only gives a larger, and moreuseful sample size, but also lets the directional aspects of thegenerate sounds be measured.

One embodiment of this process uses the mobile devices 824 of theaudience as a musical instrument: for the band to be able to ‘play’ alllike a 20,000 key keyboard and/or a 20,000 fret guitar, for example. Inanother embodiment, the voices of the audience are individually recordedwith their mobile devices 824 as content stored on NAS 850. Sitecontroller 812 can also ‘play’ the audiences' voices to create an effectin the venue or create a recording that would include the voice of everysingle audience member on the entire tour or a subset such as one show.

In one embodiment, a component of site system 280 (e.g., sound systemcontroller 1016) can establish a connection with mobile device 824 so asto have mobile device 824 act as an amplifier and speaker (e.g., usingspeaker 962). Site system 280 (e.g., using sound system controller 1016)can divide a song (and/or any other audible signal) up into constituentparts and spread these across different groups of selected mobiledevices 824. In an example the song could be divided such that somemobile devices would play the drums, some the harmony, some the melody,some the lyric line, etc. In some embodiments, the audio effects can becombined with visual effects on mobile devices 824.

In one embodiment, part of the act of buying the ticket could includethe recording of the ticket holder's voice. The app could ask the futureattendee to state their name or for any other phrase to be state orsimply ask that the future attendee say whatever occurs to him or her.

The voices could also be transposed into a tone of variable length andkey and register, etc. During the show, the recorded voices could bemodified, combined and/or otherwise altered, then broadcast at the show.Broadcast could be using the standard show sound system and/or thedistributed approach described with FIGS. 23 and 24 above.

FIG. 25 shows an embodiment of a process 2500 for automaticallyoutputting media associated with an event to mobile devices based on thelocations of mobile devices relative to activities at the event. FIG. 26illustrates an implementation example of process 2500, using a stadiumhaving areas where activity on the field is not visible. At block 2510,the initialization and location estimate process 1200 is performed, andat block 2515, an activity is selected that is associated with an event.For example, in a football game event, the activity selected could bethe play on offense of one of the teams. On FIG. 26, item 2630represents a location of an activity that a spectator shown as havingmobile device 824-1 would not want to miss.

At block 2520, a first mobile device is selected to act as a measure ofwhether an activity is visible, and the estimated location of the mobiledevice is analyzed relative to a location of the selected activity. Atblock 2525, the estimated location of the first mobile device relativeto a location of the selected activity is analyzed. At block 2530 and2535, based on the analysis and a data model of the location of theselected activity, a determination is made as to whether the selectedactivity is viewable from the estimated location of the first mobiledevice.

In this example, mobile device 824-1 has a clear view of location 2630,but does not have a view from positions 2640 (e.g., an elevator) and2620 (e.g., a restaurant). When the determination is estimated not to beviewable, block 2545 uses sensors on mobile device 824-1 do determineambient characteristics of the location, e.g., an elevator bank and abathroom would likely be silent, while a restaurant can be loud. Inblock 2550, an assessment is made as to the different available waysthat information about the activity could be sent to mobile device 824-1(e.g., is there a descriptive audio stream, a video stream viewable onthe mobile device). In the embodiments that use site controller 812, adetermination can be made as to whether available media can be deliveredto a nearby video monitor.

In block 2555, based on the characteristics of the location, anavailable media solution is provided to mobile device 824-1, or otherusable display.

FIG. 27 shows an embodiment of a stage display sculpture resembling agiant lightbulb 2740 with a digital surface 2710 that can relayinformation 2705 from/to the crowd. An additional signal surface 2720can be provided that allows for additional messages 2730 to bedisplayed. In some embodiments, information gathered at the venue couldbe displayed on stage displays such as a person's full or partial name,a photo that was captured by embodiments, gathered text messages, etc.

Within the venue, stage displays can be controlled by a site controller812, e.g., using stage display(s) controller 1012. The stage displayscan play content stored in the site network 816 including pictures,video and sound gathered at the venue or even live. The stage displays136 could simulcast the performers, or display other recorded content,text, photos, audio, etc. The stage displays 136 include concepts forphysical scenic elements that will relay the information that istransmitted to the venue via the site network 116, for example:

-   -   A ring of LEDs surrounds the giant lightbulb or other object to        relay information from the site network 116 prior to and during        the event    -   A strip of LEDs could appear to deliver information that has        been gathered from the site network 116 during the show

FIG. 28 illustrates an example augmented reality user interface 2850 ona mobile device (e.g., 2885) that some embodiments use to provide a viewof a location (e.g., 2845) augmented by computer-generated visual and/oraudio information. In this example, a camera of a mobile device (e.g.,camera 976 of mobile device 824) is pointed at a view of the location.The mobile device and/or a remote server (e.g., site system 280) incommunication with the mobile device, utilizes information from themobile device to determine (e.g., estimate) what is in the view of thecamera. The determination can be based at whole or in part, on:

GPS location information (e.g., from location engine 937).

Cell tower location information (e.g., read using cellular 958 wirelessmodule, and analyzed using handheld controller 910).

WiFi location information (e.g., read using WiFi 952 wireless module,and analyzed using handheld controller 910).

A compass internal to the mobile device (e.g., magnetometer 938) thatcan provide heading information (e.g., relative to magnetic north).

An accelerometer internal to the device (e.g., accelerometer 932) (whichcan also be used to provide tilt information).

Gyroscopic orientation information from a gyroscope (e.g., gyroscope 934being a 2 or 3 axis gyroscope in some embodiments which can provide twoor three dimensional attitude information (e.g., pitch, roll, and yaw).

Object recognition performed by analyzing a captured image (e.g.,captured by camera 976) to identify landmarks (which may be structurallandmarks, such as walls, columns, doorways, seats, and/or may be activeor passive beacons, such as coded signs (e.g., where each sign has aunique visual code and the signs are strategically placed are columns,walls, etc.), etc.), faces, etc., and/or other information. This type ofvisual analysis is discussed further with the description of FIGS. 30-32below.

Altitude above sea level estimated by barometer 939.

In order to make the determination, some or all of the foregoinginformation may be used in combination with a 3D map of the location(which may include beacon placement location information, if such existand/or other landmark identifications and locations) and/or photographsand/or what is actually physically present in the venue as captured viaa rear-facing camera lens on the user's smart phone, PDA device, ortablet device. In some embodiments, this information is stored on themobile device (e.g., in application storage 922), and in someembodiments, this information is retrieved from a server (e.g., fromsite information storage 1037 in NAS 850 attached to site controller812).

In particular, some or all of the forgoing information may be used todetermine the device's pose (position and orientation). For example, GPSinformation can be used to determine the latitude and longitude locationof the user device, and gyroscopic orientation information can be usedto determine the lens angle with reference to ground or other referencepoint or plane. Upon receiving an indication (e.g., via the application)that the device's camera is active (capturing images), and by knowingthe user device pose, and the system can determine what is beingdisplayed on the device's display. Pose is discussed further with thedescription of FIGS. 30 and 31 below.

Mobile device 824 and/or site system 280 can also obtain informationabout people at the location (e.g., including identifiers, names andpositions in the location) and user friend information (e.g.,identifiers/names associated with a user's friends obtained from theticket system and/or a social network system data stores) which may becompared to determine where and in which seats the user's friends aresitting. The server can forward to the application information as towhere in the device display such seat and friend information are to bedisplayed. The application can then overlay onto the image captured bythe camera names, photographs, and/or seat identifiers of the user'sfriends so that the user can visually see where the user's friends arelocated using the augmented reality approaches described with FIGS.29-33 below. The augmented reality system may also receive comments,photographs, and/or videos posted by event attendees during the event,e.g., similar to 2846 shown on augmented reality display 2850.

As shown as 2846, for a given user, using the augmented realityapproaches described with FIGS. 29-33 below, embodiments can determinewho the user's friends are, and then stream the user's friends'comments, photographs, and/or videos submitted via the application, ashort messaging service interface, a social network interface, orotherwise, (and received by the system) in substantially real-time tothe user's device for display via the augmented reality user interface.

As described in detail with the description of FIGS. 30-32 below, othertypes of information may be overlaid onto the camera view, such ashighlights or other emphasis around entrances to bathrooms, concessions,other amenities, exits, the user automobile, etc. The emphasis may bevisually coded (e.g., color coded, icon coded, etc.), where differentcodes may be used to identify different features or types of information(e.g., the type of service provided by an amenity (e.g., food, abathroom, a water fountain, an automated teller machine).

In addition, the system may determine which of the user's friends havearrived at the venue based on an indication that their ticket (which maybe a physical ticket, an electronic ticket in their phone, a credit cardused to purchase the right to attend, etc.) has been scanned at thevenue, via a presence signal received from the friends' mobilecommunication devices while at the venue (e.g., GPS information providedvia a phone app hosted on the friends mobile communication devices), viaan affirmative action by the friend (e.g., by activating an “I havearrived” control via an app hosted on the friend's mobile communicationdevice), or otherwise (the system may similarly determine if the user isat the venue). Using approaches described with FIGS. 30-33 below, whenthe augmented reality device is pointing at a friend's seat, the systemmay overlay a color coded icon (e.g., color code, icon code, text code,etc.) the seat to indicate the friend has arrived (or that a friend hasnot arrived if their presence has not been detected). In addition orinstead, a list may be presented to the user via an application or webpage indicating which of the user's friends have arrived and which havenot yet arrived.

In certain embodiments, the ticket system may determine if the user'sview includes a performer, may access information regarding theperformer, and cause the accessed information to be displayed via theuser's augmented reality device displayed over image of the performer.

In certain embodiments, the ticket system may determine if the user'sview includes seats for which event tickets have not yet been purchased.Using approaches described herein, the system may identify the seats asbeing available to the user via an augmented reality indicationoverlaying the view (e.g., textually, graphically, or otherwise). Acontrol may be provided via which the user can purchase at a specifiedprice, via their device, a ticket/upgrade for the seat, which then maybe electronically delivered to their device to be displayed or otherwisecommunicated to others (e.g., to an usher) to indicate that the user hasa right to utilize the seat. Optionally, before indicating to the userthat a seat is available, the system may first determine if the seat isa better seat than the user's current seat (e.g., have a better view, iscloser to the stage or playing field), based on rankings or otherinformation stored in a database. If the seat is not better (e.g., has asimilar, the same, or lower ranking than the user's current seat),optionally the system does not identify the seat as being available tothe user.

FIG. 29 illustrates an example system 2900 for the generation ofaugmented reality images according to an embodiment of the presentinvention. In some embodiments, augmented reality image generated bysystem 2900 is similar to augmented reality image 2850 from FIG. 28, andis generated by a mobile device 824 (e.g., mobile device 824) havingcomponents similar to those illustrated in FIG. 9.

In some embodiments, positional sensors determine a geographic positionthe mobile device within a geographic area. For example, some positionalsensors 2965 (e.g., a GPS in location engine 937) can determine anabsolute geographic location, and/or some positional sensors 2965 candetermine a relative location within a bounded geographic area, e.g.,wireless modules 950 can determine a location using wireless signalgeolocation.

In an example embodiment, mobile device 824 can enable the selection ofand guidance to, a particular destination within the geographic area. Asdiscussed with the description of FIG. 9 above, different types ofinformation may be overlaid onto the camera view, such as highlights ofamenities at the geographic location (e.g., a venue) and the differenttypes of service provided by an amenity (e.g., food, a bathroom, a waterfountain, an automated teller machine). In some embodiments, thesedifferent services provided by a destination correspond to differenttypes of destinations.

In some embodiments, the geographic area is bounded, having a limitedarea, and different destinations within the bounded area can be selectedby a user interface 2947 (e.g., a user interface stored in applicationstorage 922) for highlighting by the augmented reality display. Usinguser interface 2947, for example, a type of destination can be selected,and different destinations corresponding to the selected type can beidentified, one or more of these destinations being shown on theaugmented reality display in a way similar to augmented reality display2850 from FIG. 28.

In some embodiments, the types of destinations, and information aboutthe destinations can be stored in, and retrieved from storage 2940(e.g., similar to application storage 922). In some embodiments, thisdestination information can be retrieved by using network connection2945 (e.g., using wireless modules 950 of mobile device 824) fromnetwork storage (e.g., site feature data storage 1039 in NAS 850). Insome embodiments, this information provides the name and location ofdestinations within the geographic area, and this information is used bysome embodiments to provide information using user interface 2947, andgenerate an augmented reality image the includes the destinations.

Some embodiments generate augmented reality image 2995 (e.g., acombination of a camera view and graphic highlights, similar toaugmented reality display 2850), two different images, termed herein acaptured image (e.g., captured image 2985 captured by camera 976) and anoverlay image (e.g., overlay image 2975) are combined. Broadly speaking,the overlay image overlays computer generated graphics over the capturedimage to produce augmented reality image 2995.

To overlay graphics in an overlay image accurately over features in acaptured image, the overlay image (e.g., overlay image 2975) isgenerated by an augmented reality engine 2960 coordinating the differentprocessed described herein. In some embodiments, part or all ofaugmented reality engine 2960 can be generated by handheld controller910 executing computer-executable code retrieved by storage controller920 from in code storage 926. As discussed further below, parts ofaugmented reality engine 2960 can be enabled by mobile device 824hardware components (e.g., motion coprocessor 915, 3-D Engine 916,physics engine 917, graphics processor 968, and/or other similarcomponents.

In an example, to display augmented reality image 2995, camera 976captures captured image 2985-1. It should be appreciated that, eventhough augmented reality display 2990 is discussed showing a single,still augmented reality image 2995, some embodiments can performdifferent processes described herein with full motion video, such videobeing a collection of still images, the still images making up the fullmotion video.

Continuing this example, at the time captured image 2985-1 is captured,the “pose” of mobile device 824 is estimated. In some embodiments, thiscamera pose (or just pose), includes one or more of the geographiclocation of the mobile device, the 3-D orientation of the mobile device,as well as the compass direction (e.g., as measured by magnetometer 938)and the altitude of the mobile device (e.g., as measured by barometer939.

Continuing this example, once a camera pose is determined, user input isanalyzed and storage is accessed to determine which objects should behighlighted. In some embodiments, to place points of interest properlyin overlay image 2975-1, augmented reality engine 2960 can use a virtualcamera 2955. In this example, the geographic position determined bymobile device 824 combined with the camera pose from determines thatmobile device 824 is overlooking venue 2930 and point of interest 2945-1(also termed features, and/or destination) is in the field of view. Withfurther processing it can be determined by the virtual camera 2955approach that point of interest 2945-1 should be highlighted (e.g., the“reality” of captured image 2985-1 should be augmented by graphical icon2946 in overlay image 2975-1).

Once all the highlighted features are selected, and the graphical iconsare properly placed, overlay image 2975-1 can be generated. As will bediscussed in greater detail with the description of FIGS. 30-32 below,captured image 2985-2 and overlay image 2975-2, are respectivelycaptured and generated using the same camera pose. To generate augmentedreality image 2995, captured image 2985-2 and overlay image 2975-2 arecombined, e.g., with an alignment that places highlight icons over thepoints of interest. In some embodiments, image combiner 2970 usesdifferent approaches to receive captured image 2985-1 and overlay image2975-1, and align the received images such that the graphical highlightsof image 2975-1 are aligned with their corresponding physical featurescaptured in captured image 2985-1. Example approaches to this alignmentare discussed below with FIG. 30.

This combination and alignment can also be termed “registration” in someembodiments, and the process can happen frequently, e.g., in full motionvideo embodiments.

FIG. 30 illustrates an example captured image 3010 (e.g., captured image2985-1 from FIG. 29), as captured by a camera (e.g., camera 976 ofmobile device 824 shown FIG. 9). As noted above, in some embodiments,augmented reality display 2990 shows a succession of generated augmentedreality images 2995 resulting in a full-motion video display of reality,augmented with icons attached to features on a landscape. In thisexample, image 3010 can be a still image, or a frame of a video streamdisplayed on mobile device 824 showing landscape 3060.

In this example, landscape 3060 is at a location where a user of mobiledevice 824 seeks to differentiate between two types of destinations,such example destinations being represented as simple shapes in FIG. 30(shown as rectangle 3045 and triangle 3040 on landscape 3060). In anembodiment rectangle 3045 is a representation of a concession andtriangle 3040 is a representation of a restroom. It should beappreciated that landscape 3060 is shown (as it would be on augmentedreality display 2990 (e.g., display 965 on mobile device 824)) a twodimensional representation of a three dimensional scene. To improve therendering of landscape 3060 in FIG. 30, X 3022, Y 3021 and Z 3023 axesare shown, though these need not be displayed in embodiments.

As discussed above with FIG. 29, in some embodiments, one or more typesof overlay image generation are used to generate overlay image 2975-2.One way uses the pose of a camera that captures the captured image forwhich the overlay image is to be generated (e.g., as determined bypositional sensors 2965) at the time captured image (e.g., capturedimage 2985-2) was captured. This approach, using positional sensors anda determined camera pose, is termed herein as an “inertial” approach. Asdiscussed above, a camera pose can include a combination of ageographical position (e.g., determined by location engine 937), acamera direction (e.g., determined by magnetometer 938), altitude (e.g.,as determined by barometer 939), and a three dimensional representationof the orientation of mobile device 824 (e.g., determined by one or moregyroscopes 934). It would be appreciated by one having skill in therelevant art(s), given the description herein, that the example camerapose described above can be combined with geographic locationinformation of a landscape 3060 feature (e.g., restroom 3040-1) toselect the placement of a representative graphical icon in overlay image2975-1. In some embodiments, this geographical information can bereceived as overlay data 2946-1 from storage 2940 (e.g., applicationstorage 922) or received as overlay data 2946-2 from network connection2945 (e.g., using wireless modules 950 to receive from site controller812). In some embodiments, network connection 2945 receives geographiclocation information from the points of interest themselves, e.g., aswireless transmissions of data using BLE or other technology.

In selecting the placement of the representative graphical icon in theoverlay image, some embodiments of handheld controller use one or moreof motion coprocessor 915, 3D engine 316 and physics engine 317.

Another approach to the generation of overlay image 2975-1 uses a visualanalysis of a captured image (e.g., captured image 2985-3 received ataugmented reality engine 2960 from camera 976) to identify points ofinterest in the captured image for subsequent overlay of graphicalhighlights. In some embodiments, this visual analysis is assisted bydifferent visual markers attached to points of interest. These markersare shown in FIG. 30 as marker 3050 and marker 3051, respectivelymarking concession 3045-1 and restroom 3040-2. The markers are differentfrom each other in this example, to relay an indication to augmentedreality engine 2960 of the particular type of destination they mark.Thus, in a visual analysis of scene 3010, markers 3050 and 3051 aredetected by augmented reality engine 2960 and their position in scene3010 is used to place appropriate (e.g., by destination type) graphicalhighlights in overlay image 2975-1. One having skill in the relevantart(s), given the description herein, would appreciate how a capturedimage (e.g., captured image 3010) can be analyzed to detect a visualindicator, and how this indicator can be used to overlay graphicalhighlights on features to which the markers are attached. This approachcan be termed herein a “visual approach” to augmented reality.

It should be noted that a combination of visual and inertial approachescan be used to generate overlay image 2975-1. For example, only aportion of the points of interest in scene 3010 have markers to enablevisual analysis of the scene. Once overlay image 2975-1 is generated, insome embodiments, overlay image 2975-1 is relayed to image combiner 2970and combined with captured image 2980-2 for display as augmented realityimage 2995.

FIG. 31 illustrates an example overlay image 3110 generated to becombined with captured image 3110. As with FIG. 30, to improve theexplanation shown in FIG. 30, X 3022, Y 3021 and Z 3023 axes are shown,though these need not be displayed in embodiments. The term “overlayimage” is used herein for convenience, because, in some embodiments,when captured image 3010 and overlay image 3110 are combined the effectis that the graphical icons in overlay image 3110 are superimposed(i.e., overlaid) over highlighted elements in the captured image.

As discussed above with FIG. 29, in some embodiments, one or more typesof overlay image generation are used to generate overlay image 2975-2.One way uses the pose of a camera that captures the captured image forwhich the overlay image is to be generated (e.g., as determined bypositional sensors 2965) at the time captured image (e.g., capturedimage 2985-2) was captured. This approach, using positional sensors anda determined camera pose, is termed herein as an “inertial” approach. Asdiscussed above, a camera pose can include a combination of ageographical position (e.g., determined by location engine 937), acamera direction (e.g., determined by magnetometer 938) and a threedimensional representation of the orientation of mobile device 824(e.g., determined by one or more gyroscopes 934). It would beappreciated by one having skill in the relevant art(s), given thedescription herein, that the example camera pose described above can becombined with geographic location information of a landscape 3060feature (e.g., restroom 3040-1) to select the placement of arepresentative graphical icon in overlay image 2975-1. In someembodiments, this geographical information can be received as overlaydata 2946-1 from storage 2940 (e.g., application storage 922) orreceived as overlay data 2946-2 from network connection 2945 (e.g.,using wireless modules 950 to receive from site controller 812). In someembodiments, network connection 2945 receives geographic locationinformation from the points of interest themselves, e.g., as wirelesstransmissions of data using BLE or other technology.

In selecting the placement of the representative graphical icon, someembodiments of handheld controller use one or more of motion coprocessor915, 3D engine 316 and physics engine 317.

Another approach to the generation of overlay image 2975-1 uses a visualanalysis of a captured image (e.g., captured image 2985-3 received ataugmented reality engine 2960 from camera 976) to identify points ofinterest in the captured image for subsequent overlay of graphicalhighlights. In some embodiments, this visual analysis is assisted bydifferent visual markers attached to points of interest. These markersare shown in FIG. 30 as marker 3050 and marker 3051, respectivelymarking concession 3045-1 and restroom 3040-2. The markers are differentfrom each other in this example, to relay an indication to augmentedreality engine 2960 of the particular type of destination they mark.Thus, in a visual analysis of scene 3010, markers 3050 and 3051 aredetected by augmented reality engine 2960 and their position in scene3010 is used to place appropriate (e.g., for their type) graphicalhighlights in overlay image 2975-1. One having skill in the relevantart(s), given the description herein, would appreciate how a capturedimage (e.g., captured image 3010) can be analyzed to detect a visualindicator, and how this indicator can be used to overlay graphicalhighlights on features to which the markers are attached. This approachcan be termed herein a “visual approach” to augmented reality.

It should be noted that a combination of visual and inertial approachescan be used to generate overlay image 2975-1. For example, as notedabove, only a portion of the points of interest in scene 3010 havemarkers to enable visual analysis of the scene. Once overlay image2975-1 is generated, in some embodiments, overlay image 2975-1 isrelayed to image combiner 2970 and combined with captured image 2980-2for display as augmented reality image 2995.

In an embodiment, arrow 3150 highlights one graphical icon (e.g.,3140-1) of the many graphical icons in overlay image 3110. As describedwith FIG. 27 in the incorporated '292 Application, many different typesof physical features can be displayed, using embodiments discussedherein, e.g., available seats, friends at the location, concessions,exits, and/or other items useful to locate.

In one example, restrooms are available at a location with adesirability based on a combination of different factors (e.g., queuelength and distance from the mobile device). After restrooms areselected as a destination type, destinations of the restroom type (e.g.,3040-1 and 3040-2) could be identified (e.g., by site controller 812using information stored in site information storage 1037). In someembodiments, information about these destinations could also be storedon mobile device 824 (e.g., in application storage 922).

Once different destinations have been identified, some embodiments canidentify information about the destinations that could reflect on theirevaluation (e.g., queue length to access the destination). Differentexample approaches to monitoring queue length by a server (e.g., by sitecontroller 812) are described in the '292 Application. In an augmentedreality example, operating on mobile device 824, current queue lengthscould be sent from site controller 812 to mobile device 824. Based onthis information, and other information (e.g., distance from mobiledevice 824 to each destination), handheld controller 910 can evaluatethe available destinations and select one or more for highlighting(e.g., 3150) as a selected destination. In some embodiments, thisanalysis can be performed by site controller 812 and results can betransferred to mobile device 824 for display.

One having skill in the relevant art(s), given the description herein,will appreciate that the identification of a type of destination,identification of destinations of the type, identification ofinformation for evaluating (e.g., ranking) the destinations, andselection of one or more destinations to highlight can be performed atdifferent times in the augmented reality image generation processdescribed with FIGS. 29-33.

FIG. 32 illustrates a combination of captured image 3010 and overlayimage 3110. In some embodiments, once captured image 3010 is capturedand overlay image 3110 is generated the two images are combined to forman augmented reality image (e.g., augmented reality image 3210). Itshould be noted that, as shown on FIG. 32, graphical icons from overlayimage 3110 are shown over their respective physical features fromcaptured image 3010 (e.g., restroom icon—1 overlaying restroom 3040-1).

In some embodiments, to achieve the combination of captured image 3010and overlay image 3110, the two images aligned such that the graphicalhighlights (e.g., restroom icon 3140-1) are properly aligned with theircorresponding physical features (e.g., restroom 3040-1). In an exampleembodiment, this alignment is performed by image combiner 2970. FIG. 29shows captured image 2985-2 and overlay image 2975-2 subject to analignment operation.

In some embodiments, a combination of one or more types of imagealignment approaches are used to align captured image 2985-2 and overlayimage 2975-2. One type of image alignment uses the pose of camera 976(e.g., as determined by positional sensors 2965) at the time capturedimage 2985-2 was generated. As discussed above, an example camera poseincludes a combination of a geographical position (e.g., determined bylocation engine 937), a camera direction (e.g., determined bymagnetometer 938) and a three dimensional representation of theorientation of mobile device 824 (e.g., determined by one or moregyroscopes 934). It would be appreciated by one having skill in therelevant art(s), given the description herein, that the example camerapose described above can be combined with geographic locationinformation of landscape 3060 features (e.g., the location of restroom3040-1) to enable the alignment of the captured features and overlaidelements. In some embodiments, 3D engine 916 can provide 3D alignmentprocessing, and handheld controller 910 can use edge-detectionapproaches available, for example, from code storage 926.

Just as a visual analysis approach can be used to generate overlay image3110 (e.g., as described above with the description of FIG. 31), avisual analysis approach can also be used to align captured image 3010and overlay image 3110. For example, during an alignment stage, marker3051 (e.g., marking concession 3045-1) can provide an alignment point incaptured image 3010 to which concession icon 3145-1 can be attached. Itshould be noted that, as with the generation of overlay image 2975-1discussed above with FIG. 30, a combination of visual and inertialapproaches can be used to align captured image 3010 and overlay image3110. For example, only a portion of the points of interest in scene3010 have markers to enable visual analysis of the scene.

It would be appreciated by one having skill in the relevant art(s),given the description herein, that the alignment of captured image 3010and overlay image 3110 need not be exact, as some level of inaccuracy istolerable in augmented reality displays 2990, e.g., augmented realityicons can be shown near their respective physical features withoutdetracting from the value of an augmented reality display.

FIG. 33 illustrates a flowchart of an embodiment of a process 3300 forusing sensor data and location data from an augmented reality device togenerate augmented reality images. Process 3300 can be performed in partor in its entirety by the augmented reality device (e.g., mobile device824). The augmented reality device can be any device having the types ofsensors described with respect to different embodiments describedherein, including those described with

FIGS. 8 and 9 above. The geographic position can be determined usingsignals received from a GPS satellite, a WiFi access point, a celltower, a Bluetooth signal generator and/or any other radio signaltransmission. The signal may include data indicative of a location ofthe device (e.g., a satellite identifier or GPS coordinates), and/or thesignal strength may be assessed and a geographic position determinedfrom known approaches.

Process 3300 begins at block 3310, where a geographic position of anaugmented reality device is determined. The geographic position can be,for example, a geographic position on earth (e.g., using latitude andlongitude), or a position relative to a location having boundaries(e.g., reference to coordinates on a plot of land, inside a building,and/or other similar area. Geographic position can also include thedirection that the augmented reality device is oriented (e.g., inrelated to compass points). This geographic position can be determinedby use of, for example, magnetometer 939 and/or an analysis of beaconsat a location indicating the direction mobile device 824 is facing. Whena geographic location is determined within a discrete area (e.g., on ona plot of land, inside a building, in a stadium and/or other similarareas), this can be termed a geographic location within a boundedgeographic area. As used by some embodiments, geographic location alsocan include the compass direction of the mobile device.

At block 3320, a three-dimensional orientation of the augmented realitydevice is determined. Three-dimensional orientation of the augmentedreality device can be determined by position sensors on the device,including those shown in FIG. 5, e.g., gyroscope 934, and/or otherapproaches to determining the orientation of a device. Gyroscope 934 canprovide a measurement of the rotation in rad/s around each of the threephysical axes (x, y and z), and, in some embodiments, a gravity sensor(not shown) can measures the force of gravity in m/s² that is applied toa device on all three physical axes (x, y, z).

At block 3330, a type of destination is identified. Destination type canrefer to a destination in the type of bounded area described above.Example destinations within a bounded area can include geographicfeatures (mountains, rivers, and/or other feature), specific locationinside and outside buildings (e.g., entrances, exits, restrooms,specific offices, concession stands, and/or other similar features). Insome circumstances, there may be several destinations (e.g., restroom A,restroom B) of the type of destination (e.g., restrooms) at a location.The selection can be from a user, for example, based on an interest inmoving to one or more of the several destinations of the type ofdestination. In some embodiments, this selection being made using a userinterface, voice-recognition, and/or other input approaches.

At block 3340, geographic positions for a plurality of destinations ofthe type of destination are identified. Similar to the geographicpositions discussed with stage 3310 above, the geographic positions ofthe destinations can be absolute (e.g., latitude and longitude), and/orrelative to the bounded location (e.g., relative to the bounds of abounded area). The positions can be identified by retrieving informationstored on the augmented reality device (e.g., retrieved from applicationstorage 922 from FIG. 9), and/or received from a central server (e.g.,retrieved by site controller 812 from site information storage 1037).

The geographic positions of destinations can be fixed (e.g., restroomsat a location are generally fixed in the location and don't change),and/or variable (e.g., a roving food-cart can have a geographic positionthat periodically changes). Some embodiments store geographic locationsfor relatively immobile destinations and infrequently modify this storeddata (e.g., to add and remove destinations), while some embodimentsidentify geographic positions of destinations from frequently updatedsources (e.g., a server the receives GPS updated location informationfrom a food-cart). The need for updating geographic positions for usewithin block 3340 can influence where such data is stored (e.g.,frequently updated data can be retrieved from a server (e.g., sitecontroller 812), and less frequently updated information can be retriedfrom a data stored locally on the augmented reality device (e.g., fromapplication storage 922).

At block 3350, queue length information is received for the plurality ofdestinations.

Queue length information can reference an ordered list of requests touse a resource, e.g., a queue of threads waiting to use amicroprocessor. Queue length can also refer to other types of orderedresource allocation, such as people physically arranging themselves in aline to access a particular resource at a location. In this example, theresource can be the destination described herein.

With reference to the augmented reality device, queue length can bereceived from an external source (e.g., site controller 812), based on arequest for data, and/or not based on a request for data. For example,having selected a destination type (e.g., a type of resource to beaccessed, a server can retrieve and transmit to the augmented realitydevice information corresponding to the number of requests currentlypending to access the resource, e.g., for resource A, five requests arepending for access to the resource). Where the destination type is aparticular location-based resource (e.g., a concession, restroom,entrance, exit, and or other destinations at a location), queue lengthinformation can correspond to the number of people currently waiting toaccess each of the resources.

At block 3360, an image is captured using a camera of the augmentedreality device. In some embodiments, a single image is captured, and theprocesses described herein commence, and are completed with the singleimage. In some embodiments, the image captured at this block is one ofmany images that compose a dynamically updated display (e.g.,full-motion video), and the processes described with FIG. 33, and otherembodiments, can result in a full motion augmented reality display shownusing the augmented reality device.

At block 3370, an overlay image is generated. In some embodiments, asdescribed with FIGS. 29-32 above, a combination of positional data forthe augmented reality device (e.g., geographic location and threedimensional orientation) are combined with overlay data (e.g., thegeographic locations of one or more destinations) to generate an overlayimage for use creating an augmented reality image. As noted above, thisoverlay image can be a single image, or one image of a sequence ofimages used to create a dynamic, full-motion video display.

At block 3380, an augmented reality image is generated based on acombination of the captured image and the overlay image. As noted withthe description of FIG. 32 above, alignment of the images facilitatesthe proper highlighting of selected destinations.

Specific details are given in the above description to provide athorough understanding of the embodiments. However, it is understoodthat the embodiments may be practiced without these specific details.For example, circuits may be shown in block diagrams in order not toobscure the embodiments in unnecessary detail. In other instances,well-known circuits, processes, algorithms, structures, and techniquesmay be shown without unnecessary detail in order to avoid obscuring theembodiments.

Implementation of the techniques, blocks, steps and means describedabove may be done in various ways. For example, these techniques,blocks, steps and means may be implemented in hardware, software, or acombination thereof. For a hardware implementation, the processing unitsmay be implemented within one or more application specific integratedcircuits (ASICs), digital signal processors (DSPs), digital signalprocessing devices (DSPDs), programmable logic devices (PLDs), fieldprogrammable gate arrays (FPGAs), processors, controllers,micro-controllers, microprocessors, other electronic units designed toperform the functions described above, and/or a combination thereof.

Also, it is noted that the embodiments may be described as a processwhich is depicted as a flowchart, a flow diagram, a swim diagram, a dataflow diagram, a structure diagram, or a block diagram. Although adepiction may describe the operations as a sequential process, many ofthe operations can be performed in parallel or concurrently. Inaddition, the order of the operations may be re-arranged. A process isterminated when its operations are completed, but could have additionalsteps not included in the figure. A process may correspond to a method,a function, a procedure, a subroutine, a subprogram, etc. When a processcorresponds to a function, its termination corresponds to a return ofthe function to the calling function or the main function.

Furthermore, embodiments may be implemented by hardware, software,scripting languages, firmware, middleware, microcode, hardwaredescription languages, and/or any combination thereof. When implementedin software, firmware, middleware, scripting language, and/or microcode,the program code or code segments to perform the necessary tasks may bestored in a machine readable medium such as a storage medium. A codesegment or machine-executable instruction may represent a procedure, afunction, a subprogram, a program, a routine, a subroutine, a module, asoftware package, a script, a class, or any combination of instructions,data structures, and/or program statements. A code segment may becoupled to another code segment or a hardware circuit by passing and/orreceiving information, data, arguments, parameters, and/or memorycontents. Information, arguments, parameters, data, etc. may be passed,forwarded, or transmitted via any suitable means including memorysharing, message passing, token passing, network transmission, etc.

For a firmware and/or software implementation, the methodologies may beimplemented with modules (e.g., procedures, functions, and so on) thatperform the functions described herein. Any machine-readable mediumtangibly embodying instructions may be used in implementing themethodologies described herein. For example, software codes may bestored in a memory. Memory may be implemented within the processor orexternal to the processor. As used herein the term “memory” refers toany type of long term, short term, volatile, nonvolatile, or otherstorage medium and is not to be limited to any particular type of memoryor number of memories, or type of media upon which memory is stored.

Moreover, as disclosed herein, the term “storage medium” may representone or more memories for storing data, including read only memory (ROM),random access memory (RAM), magnetic RAM, core memory, magnetic diskstorage mediums, optical storage mediums, flash memory devices and/orother machine readable mediums for storing information. The term“machine-readable medium” includes, but is not limited to portable orfixed storage devices, optical storage devices, and/or various otherstorage mediums capable of storing that contain or carry instruction(s)and/or data.

While the principles of the disclosure have been described above inconnection with specific apparatuses and methods, it is to be clearlyunderstood that this description is made only by way of example and notas limitation on the scope of the disclosure.

What is claimed is:
 1. A method of using sensor data and location datafrom a device to generate augmented reality images, the methodcomprising: determining a geographic position of a mobile device, themobile device being within a bounded geographic area, the boundedgeographic area having a plurality of destinations, each destinationbeing of a type of a plurality of types of destination; determining athree-dimensional orientation of the mobile device; identifying a typeof destination of the plurality of types of destinations; transmitting afirst communication to a server, the first communication including anidentification of the identified type of destination; identifying aplurality of destinations associated with the identified type ofdestination; receiving, by the mobile device, a second communicationfrom the server, the communication including queue-length informationassociated with the identified plurality of destinations; identifying ageographic position for each of the identified plurality ofdestinations; capturing a first image using a camera of the mobiledevice; generating a second image based on: the geographic position ofthe mobile device, the three-dimensional orientation of the augmentedreality device, the queue-length information associated with theidentified plurality of destinations, and the geographic position atleast one of the identified plurality of destinations; generating anaugmented reality image based on the first image and the second image;and displaying the augmented reality image.
 2. The method of usingsensor data and location data from a device to generate augmentedreality images as recited in claim 1, wherein receiving queue-lengthinformation associated with the identified plurality of destinationscomprises receiving an estimated queue length for each of the identifiedplurality of destinations, and the method further comprises: performinga comparison to select a destination of the identified plurality ofdestinations, the comparison being based on the estimated queue-lengths;and selecting a destination of the plurality of destinations, theselection based on the estimated queue-lengths, wherein generating adigital image based on queue-length information associated with theplurality of destinations comprises generating a digital image havingthe selected destination visually highlighted.
 3. The method of usingsensor data and location data from a device to generate augmentedreality images as recited in claim 1, wherein determining a geographicposition of an augmented reality device comprises: identifying alatitude and a longitude measurement from a global positioning system(GPS) sensor in the augmented realty device; identifying a mapassociated with the latitude and longitude measurements; and combiningthe latitude and longitude measurements with the map to determine thegeographic position of the augmented reality device.
 4. The method ofusing sensor data and location data from a device to generate augmentedreality images as recited in claim 1, wherein determining athree-dimensional orientation of the augmented reality device comprisesdetermining a three-dimensional orientation of the augmented realitydevice using a gyroscope of the augmented reality device.
 5. The methodof using sensor data and location data from a device to generateaugmented reality images as recited in claim 4, wherein determining athree-dimensional orientation of the augmented reality device furthercomprises determining a three-dimensional orientation of the augmentedreality device using a compass of the augmented reality device.
 6. Themethod of using sensor data and location data from a device to generateaugmented reality images, as recited in claim 1, further comprisingdetermining movement information associated with the augmented realitydevice, wherein generating a digital image further comprises generatinga digital image based on the determined movement information.
 7. Themethod of using sensor data and location data from a device to generateaugmented reality images, as recited in claim 6, wherein determiningmovement information associated with the augmented reality devicecomprises determining movement information associated with the augmentedreality device using an accelerometer of the augmented reality device.8. The method of using sensor data and location data from a device togenerate augmented reality images, as recited in claim 1, furthercomprising identifying, based on the geographic position of theaugmented reality device and the geographic position of each of at leastone of the identified plurality of destinations, a closest destinationof the identified plurality of destinations, wherein generating thesecond image based on the geographic position of each of at least one ofthe identified plurality of destinations comprises generating the secondimage having the closest destination visually highlighted.
 9. The methodof using sensor data and location data from a device to generateaugmented reality images, as recited in claim 1, wherein identifying atype of destination of the plurality of types of destinations comprisesreceiving, by the augmented reality device, a selection of the type ofdestination.
 10. A computer-program product tangibly embodied in anon-transitory machine-readable storage medium, including instructionsconfigured to cause one or more data processors to perform actionsincluding: determining a geographic position of a mobile device, themobile device being within a bounded geographic area, the boundedgeographic area having a plurality of destinations, each destinationbeing of a type of a plurality of types of destination; determining athree-dimensional orientation of the mobile device; identifying a typeof destination of the plurality of types of destinations; transmitting afirst communication to a server, the first communication including anidentification of the identified type of destination; identifying aplurality of destinations associated with the identified type ofdestination; receiving, by the mobile device, a second communicationfrom the server, the communication including queue-length informationassociated with the identified plurality of destinations; identifying ageographic position for each of the identified plurality ofdestinations; capturing a first image using a camera of the mobiledevice; generating a second image based on: the geographic position ofthe mobile device, the three-dimensional orientation of the augmentedreality device, the queue-length information associated with theidentified plurality of destinations, and the geographic position atleast one of the identified plurality of destinations; generating anaugmented reality image based on the first image and the second image;and displaying the augmented reality image.
 11. The computer-programproduct as recited in claim 10, wherein receiving queue-lengthinformation associated with the identified plurality of destinationscomprises receiving an estimated queue length for each of the identifiedplurality of destinations, and the actions further include: performing acomparison to select a destination of the identified plurality ofdestinations, the comparison being based on the estimated queue-lengths;and selecting a destination of the plurality of destinations, theselection based on the estimated queue-lengths, wherein generating adigital image based on queue-length information associated with theplurality of destinations comprises generating a digital image havingthe selected destination visually highlighted.
 12. The computer-programproduct as recited in claim 10, wherein determining a geographicposition of an augmented reality device comprises: identifying alatitude and a longitude measurement from a global positioning system(GPS) sensor in the augmented realty device; identifying a mapassociated with the latitude and longitude measurements; and combiningthe latitude and longitude measurements with the map to determine thegeographic position of the augmented reality device.
 13. Thecomputer-program product as recited in claim 10, wherein determining athree-dimensional orientation of the augmented reality device comprisesdetermining a three-dimensional orientation of the augmented realitydevice using a gyroscope of the augmented reality device.
 14. Thecomputer-program product as recited in claim 13, wherein determining athree-dimensional orientation of the augmented reality device furthercomprises determining a three-dimensional orientation of the augmentedreality device using a compass of the augmented reality device.
 15. Thecomputer-program product as recited in claim 10, wherein the actionsfurther include: determining movement information associated with theaugmented reality device, wherein generating a digital image furthercomprises generating a digital image based on the determined movementinformation.
 16. The computer-program product as recited in claim 15,wherein determining movement information associated with the augmentedreality device comprises determining movement information associatedwith the augmented reality device using an accelerometer of theaugmented reality device.
 17. The computer-program product as recited inclaim 10, wherein the actions further include: identifying, based on thegeographic position of the augmented reality device and the geographicposition of each of at least one of the identified plurality ofdestinations, a closest destination of the identified plurality ofdestinations, wherein generating the second image based on thegeographic position of each of at least one of the identified pluralityof destinations comprises generating the second image having the closestdestination visually highlighted.
 18. The computer-program product asrecited in claim 10, wherein identifying a type of destination of theplurality of types of destinations comprises receiving, by the augmentedreality device, a selection of the type of destination.
 19. A system forusing sensor data and location data from a device to generate augmentedreality images, the system comprising: a processor; and a memory incommunication with the processor for tangibly storing a plurality ofprocessing instructions for directing the processor to: determine ageographic position of a mobile device, the mobile device being within abounded geographic area, the bounded geographic area having a pluralityof destinations, each destination being of a type of a plurality oftypes of destination; determine a three-dimensional orientation of themobile device; identify a type of destination of the plurality of typesof destinations; transmit a first communication to a server, the firstcommunication including an identification of the identified type ofdestination; identify a plurality of destinations associated with theidentified type of destination; receive, by the mobile device, a secondcommunication from the server, the communication including queue-lengthinformation associated with the identified plurality of destinations;identify a geographic position for each of the identified plurality ofdestinations; capture a first image using a camera of the mobile device;generate a second image based on: the geographic position of the mobiledevice, the three-dimensional orientation of the augmented realitydevice, the queue-length information associated with the identifiedplurality of destinations, and the geographic position at least one ofthe identified plurality of destinations; generate an augmented realityimage based on the first image and the second image; and display theaugmented reality image.
 20. The system for using sensor data andlocation data from a device to generate augmented reality images asrecited in claim 19, wherein the processing instruction to receivequeue-length information associated with the identified plurality ofdestinations comprises receiving an estimated queue length for each ofthe identified plurality of destinations, and the actions furtherinclude: performing a comparison to select a destination of theidentified plurality of destinations, the comparison being based on theestimated queue-lengths; and selecting a destination of the plurality ofdestinations, the selection based on the estimated queue-lengths,wherein generating a digital image based on queue-length informationassociated with the plurality of destinations comprises generating adigital image having the selected destination visually highlighted.